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SUMMARY 


The System Status Display Study is part of the Advanced Transport Operating Systems Program spon- 
sored by the National Aeronautics and Space Administration’s Langley Research Center and is directed 
toward the development of advanced display information for the flight deck operations of future com- 
mercial aircraft. The system status display (SSD) is an electronic display system which provides the crew 
with an enhanced capability for monitoring and managing the aircraft systems and which may be used in 
a standalone mode or in conjunction with automated system control. The benefits of this system include 
improving the crew’s interaction with the aircraft systems, improving the reliability of crew and system 
operations, and ensuring efficient use of flight deck workspace. 

This report presents the results of the second part of the study. The objective was to evaluate alternative 
design concepts developed in the first part. The evaluation was by flight simulation in a fixed-base flight 
deck simulator. Design issues included: 

1 . Display Format — Should the status information be presented in a pictorial format or alpha- 
numeric text? 

2. System Control — Are multifunction controls collocated with system status information better 
than dedicated controls separated spatially from status information? 

3. Procedural Information — Does the integration of procedural information with system status infor- 
mation improve crew performance over the use of paper checklist with the status information? 

The SSD consisted of computers, a color CRT, a display control panel, system interfaces with the 
systems, and either dedicated or multifunction control panels. The SSD was used in conjunction with a 
master caution and warning system based upon the guidelines proposed by the Alerting Systems 
Standardization Study.* The simulation implemented models of the SSD, alert, fuel, and hydraulic 
systems as well as the aerodynamics and equations of motion of a large-body transport aircraft. 

Twelve pilots with extensive commercial jet transport experience participated in the study. The simula- 
tion trials were conducted with a single pilot per trial. The other pilot was assumed to be incapacitated to 
simulate a high workload. While the pilot manually flew an approach pattern, system malfunctions 
caused alerts. The pilot was required to respond to the alert by reconfiguring the system. 

Performance measurements consisted of (1) the time required to acknowledge the alert, (2) the time 
required to complete the abnormal procedures, (3) the number of discrete steps to complete the proce- 
dures, and (4) the number of errors committed. Flight performance and the amount of flight control 
activity were measured to determine the effect of the experimental conditions on the primary flight task. 

A modified Cooper-Harper rating scale was used to measure subjective workload. 

With pictorial displays, significantly less time was required to complete the abnormal procedures, and 
there was a lower workload score. It was concluded that pictorial displays are superior to text for the 
presentation of system status information. The multifunction control interfaces collocated with the 
status information had shorter completion times and lower workload scores than the dedicated 
overhead panel. It was concluded that the collocation of the controls with the status information was the 
primary reason. 

The interactive procedure information had shorter completion times, less discrete steps, and fewer 
errors than the paper checklists. One drawback to the use of interactive procedures was that the pilots 
tended to react only to the procedures without full comprehension of the problem or the consequences 
of their actions. The experimental conditions had little impact upon the primary flight activity. 
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INTRODUCTION 


BACKGROUND 

Current transport aircraft use multifunction color CRTs to display engine and aircraft status informa- 
tion. The first study^ identified principles and guidelines to be used in the design of display formats for 
the management and control of aircraft systems. A system engineering approach was used to identify 
and classify information requirements for various aircraft systems according to flight phase. This 
approach permitted the identification of information based upon the crew’s requirements for decision 
making and performing actions. The information was partitioned into four distinct classes: (1) identi- 
fiers, (2) descriptors, (3) status, and (4) instructions. This classification helps identify information 
sources and the grouping of that information. 

One issue that remained was the display format. Should the display format be pictorial or alphanumeric 
text? Several studies have developed pictorial schematics for aircraft systems’ displays. ^ t.4.5 jhese 
studies were of conceptual designs and none was concerned with comparative evaluations of other con- 
cepts or current displays. The first operational transport aircraft to use pictorial schematics and 
checklists on CRTs for the aircraft systems is Airbus Industries’ A310. These aircraft are in active use 
which proves the concepts are feasible. However, it remains to be shown that pictorial schematics are 
improvements over other display concepts, including alphanumeric text. 

A review of the cognitive literature implies that pictures are better than words for aircraft systems’ 
displays. Pictures are recognized faster than words, but it takes longer to name pictures than words. ^ 
This may be due to the memory coding of pictures and words. Pavio^'® proposed that pictures are stored 
in direct memory which has a close relationship to the sensory experience that created the image. Words 
are stored in phonemes (speech utterances) or the visual representation of phonemes. Communication 
or comparison of pictures or a sensory experience to words is by way of the short-term working memory. 

Another factor is the retention of an aircraft system in long-term memory. Chase^ suggests that the 
memory of a system is an abstraction of the spatial and the functional relationships of objects to one 
another. Egam and Schwartz in their study of circuit diagrams concluded that experts recall patterns 
based on their conceptual knowledge of a circuit’s function. These studies suggest that the crew reten- 
tion of an aircraft system in long-term memory would be similar to the functional schematic. They fur- 
ther suggest that less mental steps are required to extract information from a pictorial schematic than 
from a word listing. 

Another issue for the SSD concept is the control interface with the aircraft systems. Are multifunctional 
controls acceptable or should dedicated controls be retained? If dedicated controls are used, they must 
be separated from the multifunction display. In transport aircraft, the only reasonable location for dedi- 
cated controls is in the overhead panel, and the only reasonable location for the SSD is in the center 
instrument panel. Multifunction controls, (e.g., line select keys or touch panels) can be collocated with 
the SSD. This confounds the issue; if multifunction controls are used, they can be collocated with the 
SSD while dedicated controls must be separated spatially. 

Two options for the multifunction controls were considered for this study: (1) line select keys, and (2) a 
touch panel overlay. Line select keys are mounted on the CRT bezel with a line pointing to the com- 
ponent that they activate. These are currently being used with flight management computer’s control 
display units. The problem with these keys is that they are separated from the displayed information. 
With a touch panel overlay, the control and information source are coincident. Although touch panels 
have advantages, there are also some disadvantages. '* These include the virtual area and separation of 
the keys on the display, separation of the touch panel from the phosphor surface of the CRT which 
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causes a parallax problem, the requirement for feedback other than tactile, the smudging of the display 
surface, and vibration caused by normal aircraft operation. 

Another potential control interface is voice-activated controls. The primary advantage of voice control 
is that it does not require hand coordination and movement. It requires speech utterances so that the 
same problems with alphanumeric text will occur with voice control. Other problems are recognition 
reliability and transmission time. Because of these drawbacks, voice control was not considered in this 
study. 

A separate issue involved with aircraft systems control is automation. With the new-generation flight 
decks such as those on the Airbus A320 and the MD-1 1, on-line monitoring and reconfiguration are 
being automated. This eliminates the need for the crew to monitor and control the systems; the crew only 
provides a backup when the automated system fails. This has raised concerns about flight safety; e.g., 
the ability of the crew to recognize and respond to failures of the automatic system and also, how the 
crew can maintain its skill level to provide this manual backup. ‘‘♦The issues include what information 
should be provided for crew monitoring of the automated functions and how information should be 
provided for manual backup. 

SYSTEM STATUS DISPLAY CONCEPT 

A system status display (SSD) was developed^ to evaluate these issues in a simulation study. The SSD 
consisted of computers, color CRT displays, display control panels, I/O interfaces with the aircraft 
systems, and control interfaces for systems control. A block diagram of the system is shown in Figure 1 . 



FIGURE 1. SYSTEM ARCHITECTURE OF THE SYSTEMS STATUS DISPLAY. 
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Each aircraft system may have its own independent control units that provide fault monitoring, detec- 
tion, and correction. The prioritization and display of alerts were provided by a master caution and 
warning (MCW) system. The MCW consisted of redundant computers, the MCW annunciators, aural 
tones, and the alert display. The MCW was designed according to the guidelines set forth by th FAA 
Alerting Systems Standardization Study.’ 

The SSD computer generates status display formats, stores and presents procedures, monitors crew 
actions, and prompts the crew. The system provides the crew with feedback on the results of its actions 
and annunciates changes in aircraft status and operating characteristics. The SSD provided information 
for operation of the aircraft systems both when the primary mode of operation was manual or when 
manual operation was the backup to automated system control. 

Using the SSD, the following issues were addressed: 

1 . Display Format — Should the aircraft status information be presented in pictorial schematics or 
alphanumeric text? 

2. System Control — Are multifunction controls with system status information better than 
dedicated controls separated spatially from status information? 

3. Procedure Information — Does the integration of procedural information with system status infor- 
mation improve crew performance over the use of paper checklist with the status information? 

The objective of this study was to use the SSD on a fixed-base simulator to resolve these issues. Both 
objective performance measures and pilot opinion were used to evaluate the results. 
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METHOD 


p„ .ORIGINAL PAGE IS 

OE POOR QUALITY 

Twelve pilots from Douglas Aircraft Company’s Flight Operations department participated in this 
study. These pilots averaged 6,800 hours in commercial jet transport aircraft and all but one had more 
than 500 hours in DC- 10 aircraft. The balance of their flight time was in DC-8, DC-9, MD-80, 727, C-5A 
and C-130 aircraft. The pilots’ mean age was 57 years, their median age was 58, and their ages ranged 
from 33 to 67. 

EXPERIMENTAL DESIGN 

A full factorial repeated-measures design was used for the two display formats, the three control inter- 
faces, and the two procedural presentations. The pilots were divided into two groups. One group 
received the interactive procedures in the first block of trials and the paper checklist in the second block. 
The other group received the opposite. The display formats and the control interfaces were counter- 
balanced within the block of trials so that each pilot received a unique sequence of conditions. There 
were two trials for each of the experimental conditions. Within each trial, three non-normal events 
occurred. These events prompted advisory, caution, and warning alerts requiring the pilot response 
while flying the aircraft. 

SIMULATOR 

A fixed-base cockpit simulator, shown in Figure 2, was used for the evaluation. This is a modified DC- 10 
cockpit with computer-driven displays, a programmable McFadden center-stick controller, rudder 



FIGURE 2. FIXED BASE COCKPIT SIMULATOR. 




5 


HI6I! ^ mTUmWAUY blakk 


pedals, a throttle quadrant, slat/flap controls, and a landing gear handle. The primary flight instru- 
ments are electromechanical. Two 8-inch color CRT displays were mounted in the center instrument 
panel for the SSD, the engine, and the alert displays. The aircraft and system models are provided by a 
DEC VAX 8600 computer. The VAX 8600 was linked to a satellite computer that serves as an input/out- 
put interface to the controls, the instruments, and the visual terrain model. The SSD model and displays 
are provided by the VAX 8600 linked to a Vector General graphics system that drives the two Xytron 
CRT displays. The graphic system is a calligraphic or stroke written system. 

The simulation consists of several models which interact with each other as shown in Figure 3. The 
equations of motion, the aerodynamics, the controls, and the engine models are based on a two-engine 
DC-10. The fuel and hydraulic system models are simplified operating models that provide system status 
information for the SSD and inputs for the engine and control models. 

The fuel model supplies manifold pressure for the engines and the consumption of fuel depends on their 
thrust settings. Loss of fuel pressure would produce a caution alert but would not cause the engine to 
flame out. The hydraulic model supplies manifold pressure for the hydraulic actuators on the control 
surfaces. The loss of hydraulic pressure in one of the manifolds results in the loss of the actuators sup- 
plied by the manifold which may affect the controllability of the aircraft. The state of the fuel and 
hydraulic systems depend upon the switch settings and system malfunctions introduced as part of the 
event scenario. Descriptions of these systems are contained in Appendix A. 

STATUS DISPLAY SYSTEM 

The SSD is the left CRT display in the center instrument panel (Figure 2). It has multiple pages which 
contain phase-of-flight and system-status information. The selection of the pages depends on the phase 
of flight, an alert, or pages can be manually selected via the SSD control panel located in the pedestal. 
The information on the pages depends on the status of the systems. The page formats include phase-of- 
flight pages and system pages. The system pages consist of formats with varying levels of detail. 

The SSD control panel, shown in Figure 4, is located on the left side of the pedestal aft of the throttle 
quadrant. This panel consists of backlighted pushbutton switches. The crew can select either the phase- 
of-flight, the fuel, the hydraulic, or the flight control surface pages. Page advance and backpage swit- 
ches are provided to select the alternate pages within a system. 

MASTER CAUTION AND WARNING SYSTEM 

The alerting system uses the guidelines developed by the FAA Alerting System Standardization Study. ' 
It consists of a MCW annunciator on the glareshield, aural tones, and an alert display that occupies the 
left portion of the right CRT in the center instrument panel. The alert display contains text annuncia- 
tions of the alerts in their associated color (warning — red, caution — amber, and advisory — cyan) and 
ordered according to alert level. The alerts are ordered chronologically within an alert level with the most 
recent at the top of the list. The syntax is the system name, the specific subsystem, and the nature of the 
problem. The sequence of events depends on the alert level as follows: 

Warning 

When a warning occurs, the master warning annunciator on the glareshield turns on and the warning 
tone sounds. The warning tone is an alternating high- and low-frequency tone lasting one second and 
sounds every three seconds until the pilot cancels it with the annunciator switch. A red alert message 
appears on the alert display and remains for the duration of the problem. The associated system page 
appears on the SSD until the problem is resolved or the pilot manually selects another page. 
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FIGURE 3. SYSTEM DIAGRAM OF THE SSD SIMULATION. 





















FIGURE 4. SSD CONTROL PANEL LOCATED IN CENTER PEDESTAL. 


Caution 

When a caution occurs, the master caution annunciator on the glareshield turns on, and a steady tone 
that lasts for one second sounds; the tone is repeated every 10 seconds until the pilot cancels it with the 
annunciator switch. An amber alert message appears below any warnings on the alert display which 
remains displayed until the problem is corrected. The system page associated with the alert is displayed 
until the problem is resolved or the pilot manually selects another page. 

Advisory 

When an advisory occurs, a single tone sounds which is similar to a chime and a cyan alert message 
appears on the alert display below the caution messages. The existing system page remains on the SSD 
until the crew selects the appropriate page. 

EXPERIMENTAL VARIABLES 

The issues enumerated in the INTRODUCTION were the display format, the system control interface, 
and the presentation of procedural information. The following variables were selected for the evalua- 
tion. 

System Display Format 

Two format conditions were selected: (1) pictorial, and (2) alphanumeric text. Examples of these for- 
mats are shown in Figures 5 and 6 for the hydraulic system. The formats for the fuel, hydraulic, and con- 
trol surfaces are presented in Appendix B. The following general rules were used to develop these for- 
mats. 

Pictorial — The overall structure of the display is a schematic of the system and is representative of the 
physical orientation in the aircraft and the functional relationship of the systems and their elements. 
Pictographs are used for the components and parallel lines connecting the pictograph elements are mani- 
folds. The fill between these lines represents the operational state of the system. If empty, the system is 
off; if filled green, the system is operating normally; and if filled in an alert-level color, the system is non- 
normal. Alphanumerics are used as labels. Quantitative information is shown by analog scales as well as 
numerics. 
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FIGURE 5. PICTORIAL SCHEMATIC OF HYDRAULIC SYSTEMS PAGE 
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FIGURE 6. ALPHANUMERIC TEXT OF HYDRAULIC SYSTEMS PAGE 
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Alphanumeric Text — The names of the system components are used to identify them. As in the pic- 
torial schematic, the names are grouped according to their physical orientation as well as the functional 
relationships. 

Adverbial phrases are used to describe the status of a system component. The color of the adverbial 
phrase indicates whether it is off (grey), operating normally (green), or is in an abnormal state (red, 
amber or cyan). Numerics are used to show quantitative information. 

System Control 

Three methods of controlling the fuel and hydraulic systems were provided: a dedicated overhead panel, 
a touch panel overlay on the SSD, and line select keys mounted on the SSD bezel. 

Dedicated Overhead Panel — The overhead panel consists of a backlighted panel with split-legend push- 
button switches. The switches are laid out in a schematic diagram with flow lines representative of mani- 
folds connecting the switches. A diagram of the overhead panel is shown in Figure 7. The switch shows a 
lighted flow line if it is operating normally, unlighted if it is off, and an annunciation if it is in a non- 
normal state. Also, low system pressure annunciations are provided. 


HYD 


AUX ENG 1 ENG 1 ENG 2 ENG 2 AUX RAT 



FUEL 


XFER XFER XFER 



FIGURE 7. DEDICATED OVERHEAD PANEL FOR FUEL AND HYDRAULIC SYSTEMS. 
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Line Select Keys — The bezel-mounted keys consist of illuminated pushbutton switches. Dashed lines 
are provided on the SSD to show which key is associated with what component on the status display. Fig- 
ure 8 shows an example of the line select keys with the fuel schematic. Feedback is provided by tactile 
means and by a change in component status on the SSD. 



FIGURE 8. LINE SELECT KEYS ASSOCIATED WITH THE FUEL DISPLAY. 


Touch Panel — A membrane touch panel manufactured by Elographics, Inc. was mounted over the 
SSD. The virtual area of the switch was a half-inch square centered over the component symbol or alpha- 
numeric text. Feedback was provided by a half-second tone and a change in component status on the 
SSD. 

Procedure Information 

The procedure information consisted of the identification of the component to be changed and the 
action to be taken. Two methods of presenting this information were evaluated: (1) interactive pro- 
cedures on the SSD, and (2) the use of a paper checklist. 

Interactive Procedures — The procedure format depended upon the display format. If the SSD format 
was pictorial, the adverb associated with the action and an arrow pointing to the component was 
displayed on the SSD. If there were more than one action required, a list of actions would appear on the 
left side of the display . For the alphanumeric text , the component name and the action adverb were listed 
at the top of the display. These two presentations are illustrated in Figure 9. The color of the procedures 
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FIGURE 9. INTERACTIVE PROCEDURES FOR THE PICTORIAL AND TEXT DISPLAYS. 


12 






is magenta and the current procedure is brighter than the remainder. If the crew member performs the 
procedure, it is deleted from the display. If the fault is not corrected or an incorrect action occurs, new 
procedures appear depending upon the system state. 

Paper Checklist — The other method was a paper checklist equivalent to the abnormal checklist used in 
current commercial transports or the quick reference handbook. An example of the checklist is shown in 
Figure 10. The entire checklist used in this study is presented in Appendix C. For each fault there is a list 
of procedures where the components and the action adverbs are listed. The checklist has decision 
branches where the crew must determine the state of the system or aircraft before deciding which branch 
to take. 

Procedure 

The pilots were scheduled for three 3-hour sessions. The first session was the briefing and a practice ses- 
sion and the remaining two were data sessions. The experimenter read the briefing to the pilot and 
answered any questions. The pilot briefing is presented in Appendix D. After completing the briefing, 
the pilot was given six practice trials, one on each of the display format/ control interface combinations 
for the first procedure presentation condition. The experimenter assisted the pilot in performing the sys- 
tem functions to accelerate the pilot’s learning. In the second session, the pilot was given two trials on 
each of the six conditions for the first procedure presentation condition. In the third, he received two 
trials on each of the six conditions for the second procedure presentation condition. After completing all 
three sessions, the pilots filled out the questionnaire that is contained in Appendix E. 


FUEL SYSTEM PRESSURE LOW (C) 


FORWARD PUMP ON 

IF PRESSURE REMAINS LOW. 

1&2XFEED OPEN 

1&2XFER CLOSE 

FUEL QUANTITY MONITOR 


FUEL CONFIGURATION (A) 

IF FUEL IN AUXILIARY TANK. 

AUXILIARY PUMP ON 

LOTKXFER VALVE OPEN 

HI TK XFER VALVE CLOSE 

WHEN BALANCED. 

HI TK XFER VALVE OPEN 

NO FUEL IN AUXILIARY TANK 

HITKFWDPUMP ON 

HI TK XFER VALVE CLOSE 

HITKXFEED OPEN 

WHEN BALANCED. 

HITKXFEED CLOSE 

HITKFWDPUMP OFF 


1 HYO SYSTEM PRESSURE (C) 

ENG PUMP OFF 

AUX PUMP - OFF 

MTR PUMP OFF 

CONTROLLABILITY REVIEW 


APPROACH 

FLAPS/SLATS EXTEND 

IF FLAPS/SLATS 00 NOT MOVE, 

AIRSPEED REDUCE 

FLAPS/SLATS 15/EXT 

IF FLAPS/SLATS DO NOT MOVE. 

O/RETLOG PERFORM 

IF ONLY FLAPS MOVE. 

FLAPS/SLATS 22/EXT 

22/RET LOG PERFORM 

OTHERWISE. 

FLAPS/SLATS 35/EXT 

IF SLATS RETRACT. 

FLAPS/SLATS - 22/EXT 

22/EXT LDG — PERFORM 

OTHERWISE. 

35/EXT LDG PERFORM 

2 HYD SYSTEM PRESSURE (C) 

ENG PUMPS OFF 

CONTROLLABILITY REVIEW 

APPROACH 

FLAPS/SLATS EXTEND 

IF FLAPS DO NOT MOVE. 

0/EXT LDG PERFORM 


3 HYD SYSTEM PRESSURE (C) 

IF QUANTITY IS OK. 

RAM AIR TURBINE UNLOCK 

OTHERWISE. 

ENG PUMP OFF 

AUX PUMP OFF 

MTR PUMP OFF 

CONTROLLABILITY - — REVIEW 

APPROACH 

FLAPS/SLATS EXTEND 

IF FLAPS/SLATS DO NOT MOVE. 

AIRSPEED REDUCE 

flaps/slats 15/EXT 

IF FLAPS/SLATS DO NOT MOVE. 

O/RETLOG PERFORM 

IF ONLY FLAPS MOVE. 

FLAPS/SLATS 22/EXT 

22/RET LDG PERFORM 

OTHERWISE. 

FLAPS/SLATS 35/EXT 

IF SLATS RETRACT, 

FLAPS/SLATS 22/EXT 

22/EXT LDG PERFORM 

OTHERWISE. 

35/EXT LDG PERFORM 


FIGURE 10. EXAMPLE OF PAPER CHECKLIST. 
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With a two-person flight crew, the pilot not flying operates the aircraft systems. However, for this 
evaluation it was assumed that the first officer was incapacitated, and the captain flew the aircraft and 
operated the systems. This simulates the worst-case condition and increases the pilot’s workload. The 
pilot flew the aircraft with manual throttles and the flight director in moderate turbulence. The 
simulator did not have glareshield controls so that the pilot could not enter the heading, vertical-speed 
and altitude-select values. The experimenter, on cue from the computer program, acted as the ATC 
operator and gave the pilot vectors and clearances. Ten seconds after the computer cued the experi- 
menter, the select values were entered into the guidance commands by the computer program. 

The initial conditions simulated an altitude of 7,000 feet, a heading of 360 degrees, and an airspeed of 
220 knots. The experimenter instructed the pilot to descend to 4,000 feet and slow to 190 knots. When 
the aircraft leveled off at 4,00u feet, the pilot was instructed to turn either to 330 or 030 degrees. After 
completing the turn, the pilot was instructed to descend and maintain l,500feet at 170 knots, and he was 
cleared to intercept the ILS. The trial ended at the time the pilot intercepted the glideslope. The total time 
of the trial was approximately nine minutes. 

Three non-normal events were inserted into the scenarios at given time intervals. The time intervals were 
from 75 to 200 seconds for the first event, 200 to 350 seconds for the second, and 350 to 500 seconds for 
the third. These time intervals were selected so that one event would not interfere with the next event, 
and the pilot would not anticipate the exact time the event would occur. The event types and the alert 
levels were confounded with the experimental conditions. Also, the alert levels were nested within the 
event types and there was an unequal number of types for each alert level. The different non-normal 
events are listed in Table 1 . There was a total of 18 scenarios with different combinations of non-normal 
events. For each condition, a pilot would receive one of the first six scenarios on his first trial and one of 
the last twelve on the second trial. Therefore, the first six event scenarios were repeated twice per pilot 
and the last twelve only once. Two of the first six-event scenarios contained three advisories, three 
scenarios contained two advisories and one caution, and one scenario contained one advisory and two 
cautions. For the last twelve scenarios, three scenarios contained two advisories and one caution, two 
scenarios contained one advisory and two cautions, and six scenarios contained one advisory, one 
caution and one warning. Therefore, each pilot received a total of 42 advisories, 26 cautions, and 6 warn- 
ings. The order in which the pilots received the event scenarios was counterbalanced between the pilots. 

When an alert occurred, the pilot was instructed to cancel the MCW annunciator, if necessary, and iden- 
tify the alert on the alert display. He would call up the system page on the SSD, if necessary, to identify 
and recognize the fault. If he had the interactive procedures, he would identify the corrective action on 
the SSD and perform the actions with the system controls. If he had the paper checklist, he would look 
up the fault, identify the actions on the checklist, and perform the actions. After performing the actions, 
he would verify the system status using the SSD. If he had lost a hydraulic system, he was instructed to 
call up the control surface status page to review the operability of the aircraft. 

Performance Measures 

After completing each trial, the pilot was asked to rate workload involved in the task. A modified 
Cooper-Harper rating scale referred to as the Bedford scale‘s was used for workload ratings. The 
reasons for selecting this scale were ease of application and pilot familiarity with the Cooper-Harper 
handling quality rating scale. In addition, the modified Cooper-Harper rating scale appears to be sen- 
sitive to differences in pilot workload. 
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TABLE 1 

NONNORMAL EVENTS 


ADVISORY ALERTS 

1. FUEL CONFIGURATION — TANK IMBALANCE WITHOUT FUEL IN THE CENTER TANK AND MAY OCCUR WITH 
EITHER WING TANK. THIS SCENARIO STARTS WITH ONE WING TANK WITH LESS FUEL THAN THE OTHER AND 
WITH THE LOW TANK ENGINE BURNING FASTER THAN THE OTHER TO CREATE AN IMBALANCE. THE PILOT IS 
REQUIRED TO TRANSFER FUEL FROM ONE TANK TO THE OTHER AND STOP TRANSFER WHEN THEY ARE 
BALANCED. 

2. FUEL CONFIGURATION - TANK IMBALANCE WITH FUEL IN CENTER TANK AND MAY OCCUR WITH EITHER 
WING TANK. THIS IS SIMILAR TO NO. 1 EXCEPT THERE IS FUEL IN THE CENTER TANK AND THE CENTER TANK 
PUMPS ARE OFF. IT REQUIRES THE PILOT TO TURN ON A CENTER TANK PUMP AND OPEN THE APPROPRIATE 
WING TANK FILL VALVE. 

3. HYDRAULIC SYSTEM 1 OR 3 ENGINE PUMP PRESSURE - THIS MAY OCCUR WITH THE ENGINE PUMPS ON 
SYSTEMS 1 AND 3 AS LONG AS THE REVERSIBLE MOTOR PUMP IS ON. CORRECTIVE ACTION IS TO TURN THE 
MOTOR PUMP ON AND THE AFFECTED PUMP OFF. 

4. HYDRAULIC SYSTEM 2 ENGINE PUMP 2 PRESSURE — THIS OCCURS IF ENGINE 1 PUMP IS ARMED, AND THE 
ONLY ACTION REQUIRED IS TO TURN OFF ENGINE PUMP 2. 

5. HYDRAULIC SYSTEM QUANTITY — THIS MAY OCCUR IN ANY OF THE THREE HYDRAULIC SYSTEMS. CORREC- 
TIVE ACTION IS TO TURN THE AFFECTED PUMPS OFF, BUT EVEN SO, IT IS FOLLOWED BY LOW SYSTEM 
PRESSURE (CAUTION). 

CAUTION ALERTS 

6. FUEL LOW PRESSURE — THIS IS CAUSED BY LOW AFT PUMP PRESSURE WITH FORWARD PUMP NORMALLY 
OFF AND CAN OCCUR WITH EITHER WING TANK. CORRECTIVE ACTION IS TO TURN THE FORWARD PUMP ON. 

7. FUEL LOW PRESSURE — THIS IS WITH BOTH FORWARD AND AFT PUMPS FAILED AND MAY OCCUR WITH 
EITHER WING TANK. CORRECTIVE ACTION IS TO CROSSFEED FUEL. 

8. HYDRAULIC SYSTEM FAILURE — THIS MAY OCCUR WITH ANY OFTHETHREE SYSTEMS. IT MAY BECAUSED BY 
LOW QUANTITY OR THE LOSS OF ALL PUMPS ON A SYSTEM. THE ONLY CORRECTIVE ACTION IS TO TURN OFF 
THE AFFECTED PUMPS. LOSS OF A HYDRAULIC SYSTEM MAY AFFECT THE CONTROL RESPONSE OF THE AIR- 
CRAFT OR DEPLOYMENT OF A SECONDARY SURFACE. 

WARNING ALERT 

9. DUAL HYDRAULIC FAILURE — THIS MAY OCCUR WITH ANY COMBINATION OF TWO SYSTEMS. THE ONLY COR- 
RECTIVE ACTION IS TO TURN OFF THE AFFECTED PUMPS. THE CONTROL RESPONSE AND DEPLOYMENT OF 
THE SECONDARY CONTROL SURFACES IS AFFECTED. 
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A time-event record of all the discrete events was made and placed into a data file. These events included 
simulated ATC directives, alert annunciations, and all discrete actions performed by the pilot including 
secondary control actions, MCW cancellations, SSD control actions, and system control actions. This 
record was analyzed to determine the following: 

1. Alert Acknowledgment Time — The time between the alert annunciation and the MCW can- 
cellation or, in the case of advisories, the SSD page selection. 

2. Abnormal Procedure Completion Time — The time between the MCW cancellation or the SSD 
page selection and the completion of all the system actions. 

3 . Number of Discrete Actions to Complete the Procedure — The number of discrete actions the pilot 
completed to process an alert. 

4. Procedure Errors — The discrete actions on the time-event record were reviewed by the experi- 
menter who divided each event occurrence into four categories: (1) the pilot performed the pro- 
cedures according to the interactive list or the paper checklist, (2) the pilot completed the pro- 
cedures but did not follow the interactive procedures or the checklist, (3) the pilot completed the 
procedures but made errors in the process, and (4) the pilot failed to complete the procedures. 

5. RMS Tracking Errors — RMS tracking errors were measured for those parameters that were to be 
held steady for a flight segment. These included heading while the aircraft was not turning, vertical 
speed while the aircraft was descending, and airspeed. These values were measured and recorded at 
five-second intervals. A 20-second window at the time the events occurred was used to evaluate the 
tracking performance. This was based upon selecting a time window shorter than the average event 
processing time. 

6. Control Activity — The control activity included the number of times the pilot changed the direc- 
tion of the primary flight controls; i.e., elevator, ailerons, and rudder and the number of times he 
hit the pitch trim button. The total number of counts were analyzed for the same 20-second win- 
dow that the tracking errors were analyzed. 
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RESULTS 


A within-group, repeated measures Analysis of Variance (ANOVA) test was used to analyze the 
parametric measures, and the Chi Square test was used for the contingency analysis of the correct 
responses. The treatment conditions were the two display formats, the three control interfaces, the two 
types of procedure information, the order of the two trials, and the order of the two events within a trial. 

An attempt was made to determine if the event-type or the alert-level influenced performance. However, 
as mentioned in the procedure, the event-type and its alert-level were confounded with the other experi- 
mental conditions. Two separate, one-way ANOVA tests were used to analyze the measures by event- 
type and alert-level. However, alert-level was nested within event-type and there were an unequal 
number of event-types within each alert-level category. If the alert-level was significant, then separate 
ANOVAs were performed for event-types within an alert-level. If there were no significant differences 
within an alert-level, it was assumed that the alert-level contributed to the differences in performance. 

ACKNOWLEDGMENT TIME 

A summary of the significant ANOVA results for acknowledgment time are presented in Table 2. The 
complete ANOVA tables for all the experimental results are presented in Appendix F. Table 2 shows that 
the type of procedure information, the trial order, the event order and their first order interactions were 
significant. Both alert-level and event-types are significant, which may be responsible for the differences 
in the trial and event order. The average detection times for type of procedure information are shown in 
Figure 11. The differences between event-types are shown in Figure 12. There were no significant dif- 
ferences between event-types within the advisory or the caution-alert levels. Figure 12 shows that detec- 
tion time was longer for advisory-alerts than either the caution- or warning-alerts. 


TABLE 2 

SUMMARY OF ACKNOWLEDGMENT TIME ANOVAs 


VARIABLE 

DEGREES OF FREEDOM 

F RATIO 

PROBABILITY 


PRIMARY VARIABLES 




(IF P>0.05 THE VARIABLE IS NOT LISTED) 


PROCEDURE 

1,11 

13.01 

0.01 

TRIAL 

1,11 

11.99 

0.01 

EVENT 

1,11 

19.60 

0.001 

PT 

1,11 

5.33 

0.05 

PE 

1,11 

10.39 

0.01 

TE 

1,11 

8.89 

0.01 


ALERT LEVEL 



ALERT 

2,22 

12.89 

0.001 


EVENT TYPE 



TYPE 

8,88 

6.57 

0.001 


ADVISORY EVENTS 



TYPE 

4,44 

2.01 

0.10 


CAUTION EVENTS 



TYPE 

2,22 

1.20 

0.30 
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FIGURE 11. ACKNOWLEDGMENT TIMES AVERAGED ACROSS PILOTS FOR TYPE OF 
PROCEDURE INFORMATION. 



FIGURE 12. ACKNOWLEDGMENT TIME AVERAGED ACROSS PILOTS FOR EVENT TYPE. 
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COMPLETION TIME 


The ANOVA results for the completion time are presented in Table 3 which shows that display format 
and procedure information were very significant while the control interface was significant. These dif- 
ferences are illustrated in Figures 13 to 15. Figure 13 shows that, on the average, text takes 1.3 times 

TABLE 3 

SUMMARY OF PROCESSING TIME ANOVAs 


VARIABLE 

DEGREES OF FREEDOM 

F RATIO 

PROBABILITY 


PRIMARY VARIABLES 




(IF P>0.05 THE VARIABLE IS NOT LISTED) 


DISPLAY 

1.11 

21.15 

0.001 

CONTROLS 

1,11 

3.55 

0.05 

PROCEDURE 

1,11 

22.87 

0.001 


ALERT LEVEL 



ALERT 

2,22 

6.24 

0.01 


EVENT TYPE 



TYPE 

8,88 

10.04 

0.001 


ADVISORY EVENTS 



TYPE 

4,44 

14.54 

0.001 


CAUTION EVENTS 



TYPE 

2,22 

4.82 

0.05 



FIGURE 13. COMPLETION TIME AVERAGE ACROSS PILOTS FOR DISPLAY FORMATS. 
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COMPLETION TIME (SEC) COMPLETION TIME (SEC) 



OVERHEAD LINE SELECT TOUCH 

PANEL KEYS PANEL 


NOTE: BAR REPRESENTS ± 1 STANDARD ERROR OF THE MEAN 

FIGURE 14. COMPLETION TIME AVERAGE ACROSS PILOTS FOR DIFFERENT 
CONTROL INTERFACES. 





longer to process than pictorial formats. Figure 14 shows that the dedicated overhead panel takes 1 .25 
times longer to process than either the touch panel or the line select keys, and Figure 15 shows that paper 
checklists take 2 times longer to process than interactive procedures. Significant differences were obtained 
for the alert-levels, event-types, and event-types within an alert-level. Figure 16 shows the completion 
time by event-type. 



ADVISORY CAUTION WARNING 

NOTE: BAR REPRESENTS s: 1 STANDARD ERROR OF THE MEAN 

FIGURE 16. COMPLETION TIME AVERAGE ACROSS PILOTS FOR EVENT TYPE. 
DISCRETE ACTIONS TO COMPLETE PROCEDURES 

The results for the number of discrete actions are shown in Table 4. There were significant differences 
for procedure information and the trial-event interaction. The differences for procedure information 
are shown in Figure 17. The pilots required more actions to complete the procedures with the paper 
checklist than with the interactive procedures. There were significant differences for alert-level and for 
event-types within the alert-level. The number of processing steps by event-type is shown in Figure 18. 
These differences in event-type may account for the trial-event order interaction. 

PROCEDURE ERRORS 

The procedure errors were differentiated into four categories: (1) correct and according to procedure, 
(2) correct but not according to procedure, (3) correct but errors made in the procedures, and (4) incor- 
rect or not completed. A Chi Square contingency analysis was performed on the frequency of occur- 
rence of the various categories for the experimental conditions. The results of this analysis are shown in 
Table 5. The only variable that was significant was the type of procedure information. Table 6 presents 
the percentage of errors by procedure type. The interactive procedures showed a higher percentage of 
correct responses and more completions using the correct procedure than the paper checklists. A 
between-group analysis was performed to determine if the group who received the interactive procedure 
condition first had fewer errors with the paper checklists than the group who received the paper checklist 
condition first. The results are presented in Table 7. There were no significant differences between the 
two groups (Chi Square = 2.33 with 3 degrees of freedom). 
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NUMBER OF DISCRETE ACTIONS 


TABLE 4 

SUMMARY OF DISCRETE ACTIONS STEPS ANOVAs 



VARIABLE 

DEGREES OF FREEDOM 

F RATIO 

PROBABILITY 

PROCEDURE 

TE 

PRIMARY VARIABLES 
(IF P>0.05 THE VARIABLE IS NOT LISTED) 

1,11 12.89 

1,11 6.94 

0.01 

0.05 

LEVEL 

ALERT LEVEL 

2,22 

19.91 

0.001 

TYPE 

EVENT TYPE 

8,88 

14.82 

0.001 


ADVISORY EVENTS 



TYPE 

4,44 

17.90 

0.001 


CAUTION EVENTS 



TYPE 

2,22 

13.37 

0.001 


CHECKLIST INTERACTIVE 

PROCEDURE INFORMATION 

NOTE: BAR REPRESENTS ±1 STANDARD ERROR OF THE MEAN 

FIGURE 17. DISCRETE ACTIONS AVERAGE ACROSS PILOTS FOR TYPE OF 
PROCEDURE INFORMATION. 
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FIGURE 18. DISCRETE ACTIONS AVERAGE ACROSS PILOTS FOR EVENT TYPES. 


TABLES 

RESULTS OF THE CONTINGENCY ANALYSIS FOR PROCEDURE ERRORS 


VARIABLE 

CHI SQUARE 

OF 

PROBABILITY 

DISPLAY 

3.136 

3 

NS 

CONTROL 

7.446 

6 

NS 

PROCEDURE 

52.365 

3 

0.001 


TABLE 6 

PERCENTAGE OF PROCEDURE ERRORS FOR PROCEDURE TYPE 



PAPER CHECKLIST 

INTERACTIVE PROCEDURE 

CORRECT FOLLOWED 
PROCEDURE 

38 

66 

CORRECT DID NOT 
FOLLOW PROCEDURE 

16 

7 

CORRECT BUT 
ERRORS MADE 

30 

22 

INCORRECT 

16 

5 
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TABLE 7 

DIFFERENCES BETWEEN THE TWO GROUPS FOR PROCEDURE ERRORS 
ON THE CHECKLIST CONDITION 



PAPER CHECKLIST FIRST 

INTERACTIVE PROCEDURES FIRST 

CORRECT FOLLOWED 



PROCEDURE 

39.9 

33.9 

CORRECT DID NOT 

FOLLOW PROCEDURE 

12.6 

19.0 

CORRECT BUT 

ERRORS MADE 

30.1 

29.8 

INCORRECT 

17.5 

15.5 


FLIGHT PERFORMANCE 

A summary of the ANOVA tests for tracking performance are shown in Tables 8, 9, and 10. The display 
format variable was significant for the heading rms error analysis as well as a number of significant inter- 
actions. The text format showed less heading error than the pictorial format. On the other hand, the 
heading rms error for alert-level or event-type was not significant. There were no significant differences 
in the vertical speed error for the primary experimental variables. There were significant differences for 
alert-level and event-type as well as event-type within alert-level. The vertical speed error for the dif- 
ferent alert types are shown in Figure 19. For the airspeed error, there were no main effects but two 
significant interactions. There were no differences for alert-level and event-type. 


TABLE 8 

SUMMARY OF VERTICAL SPEED RMS ERROR ANOVAs 



VARIABLE 

DEGREES OF FREEDOM 

F RATIO 

PROBABILITY 


PRIMARY VARIABLES 
(IF P>0.05 THE VARIABLE IS NOT LISTED) 


(NONE) 





ALERT LEVEL 



LEVEL 

2,22 

5.17 

0.05 


EVENT TYPE 



TYPE 

8,88 

3.53 

0.01 


ADVISORY EVENTS 



TYPE 

4,44 

17.90 

0.001 


CAUTION EVENTS 



TYPE 

2,22 

13.37 

0.001 










TABLE 9 

SUMMARY OF HEADING RMS ERROR ANOVAs 


VARIABLE 

DEGREES OF FREEDOM 

F RATIO 

PROBABILITY 


PRIMARY VARIABLES 




f F P>0.05 THE VARIABLE IS NOT LISTED) 


DISPLAY 

1.11 


7.64 

0.05 

DT 

1.11 


6.09 

0.05 

CT 

2,22 


6.62 

0.01 

TE 

1,11 


5.42 

0.05 

PTE 

1.11 


6.26 

0.05 

DCPT 

2,22 


3.59 

0.05 

DCTE 

2,22 


4.73 

0.05 



ALERT LEVEL 



LEVEL 

2,22 


2.97 

0.07 



EVENT TYPE 



TYPE 

8,88 


1.16 

0.33 


TABLE 10 

SUMMARY OF AIRSPEED RMS ERROR ANOVAs 



ADVISORY 


CAUTION 


WARNING 


NOTE; BAR REPRESENTS STANDARD ERROR OF THE MEAN 


FIGURE 19. VERTICAL SPEED RMS ERROR AVERAGED ACROSS PILOTS FOR EVENT TYPES. 
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CONTROL ACTIVITY 


The data in Table 1 1 show that event-order was significant. Event-type was not significant but alert-level 
was highly significant. Also event-types within an alert-level were not significant. The control activity 
for the different event-types is shown in Figure 20, which shows that the control activity was less for the 
warning alert which was only one event-type, while little difference occurred between the other event- 
types. 


TABLE 11 

SUMMARY OF CONTROL ACTIVITY ANOVAs 


VARIABLE DEGREES OF FREEDOM F RATIO PROBABILITY 


PRIMARY VARIABLES 
(IF P>0.05 THE VARIABLE IS NOT LISTED) 

EVENT 1,11 40.21 0.001 



ALERT LEVEL 



LEVEL 

2,22 

16.78 

0.001 


EVENT TYPE 



TYPE 

8,88 

1.88 

0.07 



ADVISORY CAUTION WARNING 

NOTE: BAR REPRESENTS ±1 STANDARD ERROR OF THE MEAN 

FIGURE 20. CONTROL ACTIVITY AVERAGED ACROSS PILOTS FOR EVENT TYPES. 
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PILOT RATING OF WORKLOAD 


The workload ratings were analyzed with ANOVA, and the results are presented in Table 12. The pilots 
rated the display formats, the control interfaces, and the interaction of the two as significantly different. 
The average ratings for the display and control interfaces are shown in Figure 21. There were no dif- 
ferences between the two display formats for the dedicated overhead panel control interface, but the pic- 
torial format was rated lower for the other two control interfaces. There were no significant differences 
between the two procedure conditions. 


TABLE 12 

SUMMARY OF WORKLOAD RATING ANOVAs 



PANEL KEYS PANEL 

NOTE; BAR REPRESENTS ±1 STANDARD ERROR OF THE MEAN 

FIGURE 21. WORKLOAD RATINGS AVERAGED ACROSS PILOTS FOR DISPLAY FORMATS AND 
CONTROL INTERFACES. 




PILOT RESPONSES TO THE QUESTIONNAIRE 

The pilots were asked to give their subjective preference for the treatment conditions as well as specific 
questions about the display formats, the controls, and the procedures. Table 13 shows the results of their 
preferences. It also shows that all the pilots preferred the pictorial formats and the interactive procedures 
and the majority preferred the touch panel. 

The pilots responded to the other questions as follows: 

1 . Color is used to indicate the alert level of the faulted components; is this an appropriate use of color? 


Color is satisfactory 80 

Cyan is inappropriate as an advisory 

(usually indicates non-normal status) 10 

Use color to differentiate partially failed 

control surfaces on the controls page. 10 

2. Do you recommend any changes to the pictorial formats? 

Format is satisfactory 60 

Should be simpler 20 

Should be more complex 20 

3. With the overhead panel,' is the system display necessary? 

Yes, it is necessary 80 

It is desirable 20 

4. With automated system control, is system information necessary? 

Yes, it is necessary 60 

It is desirable 30 

It is not required 10 


TABLE 13 

PILOT PREFERENCES 


QUESTION 

PREFERENCE 

PERCENT 

FORMAT 



SYSTEM 

PICTORIAL 

100 

CONTROL 

PICTORIAL 

90 


ALPHANUMERIC TEXT 

10 

CONTROL INTERFACE 

OVERHEAD PANEL 

10 


TOUCH PANEL 

60 


BEZEL PANEL 

20 


TOUCH OR BEZEL 

10 

PROCEDURE 

INTERACTIVE 

100 
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DISCUSSION 


DISPLAY FORMAT 

Both the objective and subjective performance measures indicate that the pictorial formats are superior 
to the alphanumeric text for system status information. The pilots had problems with the alphanumeric 
text even though the systems were similar to those of the DC- 10 and all except one of the pilots had expe- 
rience on the DC-10. In general, the pilots had no problems with the pictorial schematics. Several pilots 
mentioned that it required more time to read the alphanumeric text and associate it with their knowledge 
of the system to determine the system status while the status was readily apparent with the pictorial 
schematic. 

There are various questions involved with the development of pictorial formats for system information. 
One question is how far to go with pictorial information. Should a display format designer try to 
minimize text and use pictorial symbolics whenever possible, and when does the symbolic information 
start to become vague? Another question is the complexity of the displayed information. One of the 
guidelines for the design of the formats^ was to keep the information in a simple format. Although some 
pilots thought the formats were too simple, the majority of pilots thought that the complexity was 
satisfactory, and two pilots thought it should be less complex for easier assimilation. 

The systems displays were presented in pages. The first page was a graphic display without quantitative 
information. If the pilot wanted quantitative information, he could call up the second page. Only one of 
the events required the pilot to look at the quantitative information which increased the number of steps 
for that event. However, it probably improved the response time for the other events. It appears that 
presenting information in pages going from a general status to more detailed information is a satisfac- 
tory approach. 

Normally, when a dual hydraulic failure occurs, the pilots are required to change their flight plans and 
review the operability of the control surfaces. However, in this study the aircraft was on final approach 
and the pilot’s options were to continue the approach or abort. The scenario did not force them to look 
at the control surface page in order to fly the aircraft and they only looked at it as part of the procedures. 
Therefore, this evaluation did not test the utility of a pictorial control surface page. One pilot com- 
mented that the text was as easy to comprehend as the pictorial. Another pilot said “at this point you 
don’t need to look at the page, you just fly the aircraft.’’ One pilot commented that there was too much 
information on the page. Another pilot wanted the degree of failure color-coded rather than the alert- 
level color-coded. 

SYSTEM CONTROL 

The performance measure, event processing time, and the pilot workload rating were sensitive to dif- 
ferences in the control interface. The touch panel and the bezel-mounted keys had the same event proc- 
essing time and were rated equal in workload. The dedicated overhead panel had both a higher process- 
ing time and a higher workload. The results did not differentiate between type of control but to dif- 
ferences in the location of the control. Therefore, the critical factor, as far as the control interface, is the 
location and association with the system information. Since there were no differences between the 
number of processing steps or the error rate between the different controllers, it appears as if multifunc- 
tion controls are not detrimental to performance. 

The touch panel was preferred by the majority of the pilots. The principal reason was that the location of 
control is the same as the location of the information. With the bezel-mounted switches, mental effort 
was required to associate the switch location with the information location. 
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Pilot comments on switch activation were that the touch panel causes more problems than either the 
overhead or bezel-mounted switches. This was primarily due to the parallax problem with the virtual 
switch location, switch bounce, and the sensitivity of the touch screen. This led to turning the wrong 
component on, turning the component on-off in rapid succession, and repeated touching of the screen. 

PROCEDURE INFORMATION 

As anticipated, interactive procedures improved performance over a paper checklist. This was evident 
by the decrease in detection time, processing time, processing steps, and error rate. Although the inter- 
action of procedure information and the alert level could not be analyzed, it appears that the decrease in 
detection time was caused by a decrease in time for the advisory alerts. The pilots could easily miss an 
advisory since there was no master annunciation and the aural tone occurred once. The only remaining 
cues were the message on the alert display and the magenta procedures on the SSD (if it was on the cor- 
rect status page). These procedures would act as an additional cue to the pilot, and he would respond to 
the status display in addition to the alerting system. 

The paper checklist required more processing time due to finding the checklist, looking up the pro- 
cedure, and reading it off the checklist. Also, the information was displayed in two different formats 
with the interactive procedures. With the pictorial display, the action adverb with an arrow to the com- 
ponent was displayed, and with the text the component name was followed by the action. The latter 
required the pilot to relate the name associated with the status information. Even though there was not a 
significant interaction between display format and procedure information, the pictorial display with the 
interactive procedure had 57 percent of the event processing time compared to the alphanumeric text 
with the interactive procedures. With the paper checklist, the pictorial had 87 percent of the processing 
time compared to the text. This implies that reducing the procedure information to pictorial information 
with minimum text reduces the number of mental steps and improves performance. 

Another improvement with interactive procedures is in the error rate. This is due to(l) the display of the 
correct procedure instead of the crew relying on their memory or looking it up in the paper checklist, and 
(2) information feedback by the removal of the procedure once it has been completed. These factors 
reduce both the number of processing steps and the error rate. 

The most significant problem with interactive procedures was revealed by an observation by the experi- 
menter. The pilots responded to the procedures without diagnosing the problem. In fact, it turned out to 
be a game of trying to remove the procedure information from the display without ascertaining the 
status of the system. This would cause a problem if the interactive procedures do not cure the problem or 
provide assistance. The same problem will exist with automated systems when the pilot has to provide 
manual backup. Will the pilots have enough knowledge about the system operation to be able to respond 
correctly and in a timely manner? 

FLIGHT PERFORMANCE 

The flight performance was analyzed to determine if the experimental variables had an effect upon the 
primary flight task. There does not appear to be consistent results between the experimental variables 
and the flight performance measures. The text format shows less heading rms error, but there were a 
number of significant interactions. A possible explanation of tracking scores is that the sampling win- 
dow for the rms deviations was too small to obtain reliable scores. 

The only consistent difference in flight performance was in the vertical speed rms error where the caution 
and warning event types had consistently higher error scores than the advisories. In addition, the control 
activity, a measure of attention paid to the primary flight task, was less with the warning event type. 
These effects may be due to the type of failure rather than the alert-level since the hydraulic caution and 
warning alerts affect the controllability of the aircraft. 
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The pilot’s primary task was to fly the aircraft in a high workload situation. The pilots continued to fly 
the aircraft within reasonable limits and kept it on the approach pattern on every trial. Therefore, if the 
pilots exceeded their workload capacity, it appeared as degradation in processing the alert events rather 
than the primary flight task. 

ALERTING SYSTEM 

The performance measures were analyzed according to alert level to determine if the results concurred 
with the alerting systems standardization study. However, due to the noncentral location of the alert 
display and the emphasis of the study, the pilots tended to look at the SSD display prior to looking at the 
alert display. The advisory alerts had significantly longer detection times than the other two alerts. 
Advisory alerts only have a single tone and the text message on the alert display. Due to the high 
workload and the frequency of occurrence of the alerts, the advisory alerts were often missed or the 
pilots delayed acknowledgment. Another factor that contributed to the longer detection times was the 
difference in scoring. The caution and warnings were scored from the time the event occurred to the 
cancellation of the MCW. Advisories were scored from the time the event occurred to the paging of the SSD. 

The only other effect of alert-level appeared to be the control activity where the warning alert had less 
activity than either the advisories or the cautions. This would indicate that the pilots were paying more 
attention to the alerts than the primary flight task. However, this may be due to the nature of the warn- 
ing alert since they were all dual hydraulic failures which affected the controllability of the aircraft. 

Another observation by the experimenter was that the majority of pilots read down the list and expected 
the last alert to occur at the bottom of an alert category instead of at the top of the category. It appears to 
be natural for a person to read down a video display terminal and have the latest occurring line appear at 
the bottom of the screen. Although the pilots could learn that the last alert is at the top, they may regress, 
especially if there are other computer readouts with the latest information at the bottom of the list. 
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RECOMMENDATIONS 


As a result of this study, the following recommendations are made for future research on aircraft system 

displays and application for future cockpit design. 

1. Pictorial formats should be used to display aircraft systems information whenever possible. The 
addition of text to these displays should be kept to a minimum. Further studies are required to 
determine how complex this information should be. 

2. Even though graphic design principles 21 were used to develop the formats for this study^, there is 
a lack of empirical evidence that these principles are better than any other principles for system 
display design. Further studies are required on design principles for system display guidelines and 
validation by empirical evaluation. 

3. System controls should be located and associated with the display of system information. The 
closer these controls are to the pilot’s primary flight task, the better the performance. If the 
technical problems can be solved with the touch panels; (i.e., parallax, virtual switch size, feed- 
back, etc.), they are an acceptable method of control, since the control location is coincident with 
the status information. 

4. With automated aircraft systems, the crew should have knowledge of the system and be able to pro- 
vide manual backup. This requires the crew to have access to system status and the backup pro- 
cedures. Interactive procedures improve the crew’s ability to process the events but they become 
dependent upon this interactive information instead of learning the system. More work is required 
to determine how these procedures should be used and how to keep the crew involved in the task. 

5. Since this study found that the pilot read the bottom alert on the alert display as being the latest 
alert, the guidelines on the chronological listing of alerts from top-down that were developed as 
part of the Alerting Systems Standardization Study' should be reviewed. 
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APPENDIX A 

DESCRIPTION OF FUEL AND HYDRAULIC SYSTEMS 


FUEL SYSTEM 

The fuel model provides positive fuel pressure for feeding both engines. The consumption of fuel 
depends upon the throttle setting of the engines. Also, the engines could be set so that one would burn 
more fuel than the other to create a fuel imbalance. The fuel system consisted of three tanks: two main 
wing tanks and an auxiliary fuselage tank. Each tank had two fuel pumps where each pump had ade- 
quate flow for supplying both engines during takeoff. The fuel manifold interconnects the three tanks 
with fill valves, and crossfeed valves supply the engine manifolds. Each main tank has a float valve 
switch which will continually fill the main tank from the auxiliary tank as long as an auxiliary tank pump 
is on. The auxiliary tank pumps will turn off automatically when the tank is empty. The fuel system 
schematic is shown in Figure A-1. 

The normal configuration is to have the aft pumps on in the main tanks, a pump on in the auxiliary tank, 
the main tank fill valves armed, and the crossfeed valves closed. If a fuel imbalance occurs, a fuel 
schedule advisory occurs. If fuel is remaining in the main tank, the fill valve on the high tank is closed 
until the two tanks are balanced. If no fuel is remaining in the auxiliary tank, then the fill valve is closed 
and the crossfeed valve is open until the tanks are balanced. If there is low pressure in the engine 
manifold, a caution alert occurs. The forward pump is turned on and if the low pressure remains, the 
crossfeed valves are opened to supply fuel from either the auxiliary tank or the other main tank. 


TANK1 AUXTANK TANK 2 



LEGEND 

Q CHECK VALVE 
© PUMP 
O SHUTOFF VALVE 
FLOAT VALVE 
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HYDRAULIC SYSTEM 


The hydraulic system consists of three parallel continuously pressurized systems as shown in Figure A-2. 
Independent reservoirs supply fluid to each system. System 1 is powered by a left-engine-driven pump 
and a auxiliary pump for backup and ground operations. System 2 is normally powered by a right- 
engine-driven pump but with loss of pressure the left-engine pump is automatically turned on if armed. 
System 3 is powered by a right-engine-driven pump, an auxiliary pump for backup and ground opera- 
tions, and a ram air turbine. In addition, there is a reversible motor pump which transfers power between 
Systems 1 and 2, if there is loss of pressure in either system. 


SYS 1 SYS 2 SYS 3 



The relationship between the hydraulic system and the control surfaces is shown in Figure A-3. The loss 
of one hydraulic system will result in a reduction in the number of active spoilers, the possibility of the 
loss of a horizontal stabilizer actuator reducing the pitch trim rate, and the loss of an actuator to the 
flaps/slats resulting in a reduction in the airspeed at which they will deploy. The loss of two hydraulic 
systems will reduce the redundancy of the primary control surfaces affecting the control power, reduce 
the number of active spoilers, the loss of either the flaps or slats, and depending on which hydraulic 
system is lost, the loss of pitch trim. 

If there is low pump pressure, it will cause an advisory alert without the loss of system pressure. Crew 
actions are to clean up the system; that is, to turn on the auxiliary pump and turn off the affected engine- 
driven pump. With the loss of hydraulic system pressure, a caution alert occurs and action by the crew is 
to clean up the hydraulic system and to determine the effect on flight operation. With the loss of a second 
hydraulic system, a warning alert occurs. The action by the crew is to clean up the affected hydraulic 
system and to determine the effect upon the aircraft. 
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FIGURE A-3. HYDRAULIC SYSTEM BLOCK DIAGRAM 
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APPENDIX B 

SYSTEM STATUS DISPLAY FORMATS 



FIGURE B-1. PICTORIAL FORMAT FOR THE FUEL SYSTEM 
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FIGURE B-2. ALPHANUMERIC TEXT FORMAT FOR THE FUEL SYSTEM 
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FIGURE B-3. PICTORIAL FORMAT FOR THE HYDRAULIC SYSTEM 
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FIGURE B-4. ALPHANUMERIC TEXT FORMAT FOR THE HYDRAULIC SYSTEM 
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FIGURE B-5. PICTORIAL FORMAT FOR THE CONTROL SURFACES 
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FIGURE B-6. ALPHANUMERIC TEXT FORMAT FOR THE CONTROL SURFACES 
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APPENDIX D 
PILOT BRIEFING 

The objective of this study is to evaluate alternative display control interfaces for aircraft systems con- 
trol. Control of aircraft systems on advanced commercial aircraft will be automated. The issues are how 
the crew will interact with these automated systems and their ability to operate the systems in a manual 
backup mode. The specific issues addressed are: 

1 . How should the system information be displayed on multifunction displays? 

2. If mulitfunction controls are used, do they degrade performance as compared to dedicated controls? 

3. For manual system control, do interactive procedures on the display improve performance com- 
pared to paper checklists? 

The study will attempt to evaluate your ability to interact in both normal and abnormal situations with 
the fuel and hydraulic systems while flying an approach. The aircraft is a two-engine widebody with 
handling qualities of a DC-10. Normally, this is a two-crew operation where one pilot flies the aircraft 
and the other interacts with the system. However, we are assuming a worst-case condition where one of 
the pilots is incapacitated. 

The flight displays are electromechanical instruments. The left center CRT is the System Status Display 
and the right display is a dedicated alert display. The other portion of the right display would normally 
be occupied by engine displays^ It is assumed that these displays are interchangeable and for normal two- 
crew operations they would be reversed. The displays have been moved aft so that they are within reach 
to touch the controls. However, this does create a viewing angle and parallax problem. 

The SSD display has multiple pages which contain the following information: 

1 . Phase of flight information — These are text displays that contain information pertinent to the 
phase of flight. 

2. Hydraulic system — There are two pages: one consists of the status and the other includes quan- 
titative information with the status. 

3. Fuel system — There are two pages similar to the hydraulic system. 

4. Control surface — This contains information on the operability of the control surface as well as 
position information. 

You may select any page at any time by using the control panel which is located in the pedestal forward 
of the throttle quadrant. Normally, the flight phase page will be displayed except when there is either a 
caution or warning alert. When the latter cases occur, the appropriate systems page is displayed. The 
selection of the pages will depend on the phase of flight, an alert, or manual selection via the control 
panel. Part of the evaluation will have procedures on the pages that will interact with system state and 
actions performed by the pilot. 

You will be given two different display formats as an experimental condition: one will be pictorial 
schematics of the systems and a pictorial of the control surfaces and the other will be text containing the 
same information. Another experimental condition will be procedures on the system display that inter- 
act with the system state and the actions performed by the pilot. The other condition will be without 
these procedures and you will have to rely on paper checklists. 
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The schematics are self-explanatory. The text is structured so that the spatial relationships of com- 
ponents are the same as on the schematics. The first page of the system shows the status of the com- 
ponents and the second page adds quantitative information to the status. The control surface page 
shows a plan view of the aircraft with an outline of the control surfaces. If a control surface is operating 
normally, it is shown by a green outline. If it is inoperative, it is shown with solid fill in the appropriate 
alert-level color. If it is partially operative (i.e., with one actuator), it is shown with partial fill in the 
appropriate alert-level color. 

The color code is the following: 

1 . The alert-level colors are red — warning, amber — caution, and cyan — advisory. 

2. White is used for titles and names of the systems and the components. 

3. Green is used for systems and components that are operating normally. 

4. Magenta is used for the procedures. 

For the experimental condition where there are procedures on the system pages, the schematics w ill have 
the action and an arrow to the component when there is a non-normal condition. For the text and the 
displayed procedures that are not interactive, the component or system and the action are identified. For 
the interactive procedures they will disappear after the action is performed. The noninteractive pro- 
cedures will remain displayed until the non-normal condition goes away. 

You will be able to interact with the fuel and hydraulic systems by one of three methods, depending upon 
the experimental condition: ( 1) adedicated overhead panel, (2) a touch panel overlay of the SSD, and (3) 
bezel-mounted switches where dashed lines show which switch a component is connected, similar to line 
select keys. These inputs will allow you to control the state of the system. 

The fuel system consists of two main tanks and an auxiliary tank. Each tank has two pumps and a fill 
valve. There is a manifold between the tank with two crossfeed valves that permit total crossfeed 
capability and transfer between the tanks. The main tank fill valves have float switches which close 
automatically w hen the tanks are full. The auxiliary tank pumps will also turn off automatically when 
the tank is empty. 

The hydraulic system consists of three parallel systems. System 1 is powered by a left-engine-driven 
pump and an auxiliary pump. System 2 is normally powered by a right-engine pump but with loss of 
pressure the left-engine pump turns on if it is armed. System 3 is powered by a right-engine pump, an 
auxiliary pump and a ram air turbine. In addition, a reversible motor pump transfers power between 
Systems 1 and 3 when there is a loss of pressure in either system. 

The hydraulic system is interconnected with the control surfaces similar to the DC-10. So with the loss of 
a hydraulic system, the controllability of the aircraft is affected. 

The alert system will consist of the MCW display on the glareshield, aural tones, and the alert list on the 
right display. The alert list will contain text annunciations of the alerts in their appropriate color. There 
are three distinct aural tones for caution, warning, and advisory alerts. Pushing the MCW cancels the 
light and the aural, but the alert list remains until the alerting condition goes away. 

First, you will go through the normal start procedures. Second, you will go through the approach with a 
sample of alerts, and this will be repeated with each of control interfaces and the displays. You will have 
the chance to try more flights if you think it is necessary. 
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You will be flying with manual throttles and a flight director. The handling qualities are that of a DC- 1 0. 
You will start at a heading of 360 and a speed of 220 knots. The course pointer is aligned with the ILS. 
You will be given vectors to the ILS and approximately 10 seconds after these are given, the flight director 
and fast/slow indicator will present these commands as if you rotated the speed and heading select controls. 

Since we are attempting to measure workload with the different configurations, perform all the 
necessary normal and abnormal functions during the approach. The primary emphasis in this evaluation 
is the accuracy and the speed at which you can interact with the systems. Continue in the approach until 
you think it is unsafe to land and then initiate a go-around maneuver. Normally, the simulation will be 
terminated after you intercept the glideslope. 
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APPENDIX E 
EVALUATION FORM 


NO: DATE: 

NAME: POSITION: 

BIRTHDATE: 

WEAR GLASSES/CONTACTS FOR FLYING? YES NO 

ACTIVE FLIGHT DUTY? YES NO 

TRANSPORT EXPERIENCE: 

AIRCRAFT TYPE/MODEL POSITION NO. OF HRS DATE RATED 


PRECEDING PAGE BLANK NOT PILIfED 
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CONDITION 


RATING 
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1) Which of the two formats, pictorial schematics or text, do you prefer for the following systems information? (Please explain) 
a) Fuel and hydraulic systems. 


b) Flight control surfaces. 


2) In both formats color is used to indicate the faulted components and the alert level. Is this the appropriate use of color? 
Do you have any recommendations for other uses in this application? 


3) Do you have any suggestions for changes in the pictorial formats? 
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4) Do you have any suggestions for changes in the text formats? 


5) Which of the control interfaces do you prefer: dedicated overhead panel, touch panel or the bezel panel? (Please explain) 


6) With the dedicated control panel is the system information necessary? (Please explain) 


7) With automated control of the systems is the system information necessary? (Please explain) 


8 ) Which do you prefer: the interactive procedures or the paper checklists? (Please explain) 
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APPENDIX F 

ANALYSIS OF VARIANCE TABLES 

Acknowledgment Time 

Primary Varialbes 



SOURCE 

SUN OF 

DEGREES OF 

NEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PRUBABlLlTy 


MEAN 

44325.56314 

1 

44325.58314 

57.08 

0.0000 

1 

ERROR 

8542.277/8 

11 

776.570/1 




DIS 

22.67109 

1 

22.6/109 

0.06 

0.8039 

2 

ERROR 

3852.19164 

11 

350.19924 




CON 

605.61820 

2 

302.80910 

0.61 

0.5520 

3 

ERROR 

10909.49855 

22 

495.88630 




DC 

2626.45294 

2 

1313.22647 

1.6? 

0.2084 

4 

ERROR 

17137.75017 

22 

778.98864 




PRO 

2478.15624 

1 

2478.15624 

13.01 

0.0041 

5 

ERROR 

2095.98317 

11 

190.54392 




DP 

99.38593 

1 

99.38593 

0.14 

0.7142 

6 

ERROR 

7739.47711 

11 

703.58883 




CP 

957.17156 

2 

478.58578 

0.?4 

0.4042 

7 

ERROR 

11152.08233 

22 

506.91283 




DCP 

127.08676 

2 

63.54338 

0.13 

0.8/6/ 

8 

ERROR 

10558.38906 

22 

479.92678 




REP 

6386.86014 

1 

6386.86014 

11.9? 

0.0053 

9 

ERROR 

5861.25873 

11 

532.84170 




DR 

352.33165 

1 

352.33165 

1.16 

0.3049 

10 

ERROR 

3346.09297 

11 

304.19027 




CR 

1685.67251 

2 

842.83626 

1.68 

0.2088 

11 

ERROR 

11014.58635 

22 

500.66302 




DCR 

1348.63738 

2 

674.31869 

0,97 

0.3940 

12 

ERROR 

15262.91523 

22 

693.76887 




PR 

2367.04959 

1 

2367.04959 

5.33 

0.0414 

13 

ERROR 

4885.46350 

11 

444.13305 




DPR 

14.18527 

1 

14.18527 

0.03 

0.8688 

14 

ERROR 

5461.15098 

11 

496.46827 




CPR 

832.80051 

2 

416.40026 

0.83 

0.4507 

15 

ERROR 

11083.68990 

22 

503.80409 
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DCPR 

77*37120 

n 

38.68560 

0.07 

0.9287 

16 

ERROR 

11472*99456 

2'^ 

521.49975 




EOE 

5327.92951 

1 

5327.92951 

19.60 

0.0010 

17 

ERROR 

2990.68082 

11 

271.88007 




DE 

290.69116 

1 

290.69116 

0.92 

0.3580 

18 

ERROR 

34/3.84218 

11 

315.80383 




CE 

627.56907 


313.78454 

1.12 

0.3451 

19 

ERROR 

6180.05676 

22 

280.91167 




DCE 

372.37715 

2 

186.18857 

0.31 

0.7373 

20 

ERROR 

13251.75890 

22 

602.48904 




PE 

3775.32421 

1 

3775.32421 

10.39 

0.0081 

21 

ERROR 

3995.17738 

11 

363.19794 




DPE 

132.74110 

1 

132.74110 

0.18 

0.6814 

00 

ERROR 

8212.55619 

11 

746.59602 




CPE 

470.61734 

2 

235.30867 

0.38 

0.6851 

23 

ERROR 

13452.60961 

Z.4. 

611.48225 




DCPE 

459.80318 

2 

229.90159 

0.34 

0.7121 

24 

ERROR 

146/0.43286 

2*> 

666.83736 




RE 

2799.07142 

1 

2799.07142 

8.98 

0.0121 

25 

ERROR 

3427.57698 

11 

311.59791 




DRE 

158.25010 

1 

158.25010 

1.09 

0.3185 

26 

ERROR 

1594.78348 

11 

144.93077 




CRE 

818.42084 

2 

409.21042 

' 0.96 

0.4001 

27 

ERROR 

9424.26148 

22 

428.37552 




DCRE 

526.37750 

2 

263.18875 

0.42 

0.6622 

28 

ERROR 

13787.13625 

22 

626.68801 




PRE 

1424.84230 

1 

1424.84230 

3.02 

0.1101 

29 

ERROR 

5189.46979 

11 

471.76998 




DPRE 

383.84316 

1 

383.84316 

0.65 

0.4376 

30 

ERROR 

6508.20298 

11 

591.65482 




CPRE 

1225.18953 

n 

612.59476 

1.21 

0.3161 

31 

ERROR 

11162.93893 

22 

507.40631 




DCPRE 

207.52737 

2 

103.76369 

0.16 

0.8545 

32 

ERROR 

14412.46995 

22 

655.11227 




56 


Alert Level 


SOURCE 

SUM OF 

DEGREES OF 

nm 

F 

TAIL 


SQUARES 

KRtEDOh 

SQUARE 


PROBABILITY 

MEAN 

1390.63114 

1 

1390463114 

67.03 

0,0000 

1 ERROR 

228.22005 

11 

20.74728 



TYPE 

613.41383 

2 

306.70692 

12.85 

0.0002 

2 ERROR 

525.03206 

22 

23.86^)09 




Event Types 




SUUKUE 

SUM OF 

DEGREES OF 

KEAN 

F 

TAIL 


SQUARES 

FREEPOH 

SQUARE 


PROBABILITY 

MEAN 

5014.55895 

1 

5014.55895 

131.02 

0.0000 

1 ERROR 

420.99793 

11 

38.27254 



TYPE 

1759.85096 

8 

219.98137 

6.57 

0.0000 

2 ERROR 

2945.56061 

88 

33.47228 




Advisory Events 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

r 

TAi!. 


SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


KEAN 

5857.45694 

1 

5857.45694 

IQS. 45 

0 » O' -'iO 

1 

ERROR 

594,14320 

11 

54.01302 




TYPE 

481,61118 

4 

120.40279 

2.01 

O.IOV? 

2 

ERROR 

2632.12881 

44 

59.82111 




Caution Events 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

337.08960 

1 

337.08960 

79.10 

0.0000 

1 

ERROR 

46.87879 

11 

4.26171 




TYPE 

7.90875 

2 

3.95437 

1.20 

0.3193 

o 

ERROR 

72.30838 

22 

3.28674 
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Completion Time 

Primary Variables 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

500927.75378 

1 

500927.75378 

214.24 

0.0000 

1 

ERROR 

25720,15997 

11 

2338.19636 




DIS 

7007.57365 

1 

7007,57365 

21.15 

0.0008 

2 

ERROR 

3644.82865 

11 

331.34806 




CON 

4361.14803 

2 

2180.57401 

3.55 

0.0461 

3 

ERROR 

13511.33673 

22 

614,15167 




DC 

2878.25091 

2 

1439.12546 

1.87 

0.1784 

4 

ERROR 

16969.00362 

22 

771.31835 




PRO 

32672.31039 

1 

32672.31039 

22,87 

0.0006 

5 

ERROR 

15716.51130 

11 

1428.77375 




DP 

1000.75744 

1 

1000.75744 

1.09 

0.3189 

6 

ERROR 

10099.27190 

11 

918.11563 




CP 

2961.24911 

2 

1480.62456 

1.48 

0.2496 

7 

ERROR 

22021.57725 

22 

1000.98078 




DCP 

1069.15718 

2 

534.57859 

0.77 

0,4771 

8 

ERROR 

15363.81542 

22 

698.35525 




REP 

573.34303 

1 

573,34303 

0.72 

0.4127 

9 

ERROR 

8701.05767 

11 

791.00524 




DR 

63.00394 

1 

63.00394 

0.29 

0.5985 

10 

ERROR 

2356.89681 

11 

214.26335 




CR 

1562.61865 

') 

781.30932 

2,16 

0.1387 

11 

ERROR 

7942,77051 

'>9 

361.03502 




DCR 

158.61231 

2 

79.30616 

0.06 

0,9453 

12 

ERROR 

30940.90813 

^2 

1406,40492 




PR 

9.11184 

1 

9.11184 

0.02 

0.8982 

13 

ERROR 

5842.37308 

11 

531.12183 




DPR 

0.00944 

1 

0.00944 

0.00 

0.9974 

H 

ERROR 

9384.09282 

11 

853.09935 




CPR 

3435.13618 

2 

1717.56809 

2.04 

0.1542 

15 

ERROR 

18542.50979 

n 

842.84135 
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0.3751 


I 



DCPR 

1349.54978 

2 

674.77489 

1.03 

16 

ERROR 

14476.08533 

22 

658.00388 



EVE 

84.05988 

1 

84.05988 

0.09 

17 

ERROR 

9834.15499 

U 

894.01409 



DE 

276.25768 

1 

276.25768 

0.39 

18 

ERROR 

7725.12305 

11 

702.28391 



CE 

691.02258 

2 

345.51129 

0.38 

19 

ERROR 

20025.42143 

22 

910.24643 



DCE 

3454.15789 

2 

1727.07894 

2.46 

20 

ERROR 

15416.29870 

22 

700.74085 



PE 

34.85622 

1 

34.85622 

0.09 

21 

ERROR 

4261.39525 

11 

387.39957 



DPE 

192.03497 

1 

192.03497 

0.43 

22 

ERROR 

4925.75624 

11 

447.79602 



CPE 

1207.13110 

2 

603.56555 

0.64 

23 

ERROR 

20855.61957 

22 

947.98271 



DCPE 

1226.73089 

2 

613.36544 

1.45 

24 

ERROR 

9290.00943 

22 

422.27316 



RE 

1747.49082 

1 

1747.49082 

2.58 

25 

ERROR 

7441.04551 

11 

676.45868 



DRE 

262.68036 

1 

262.68036 

0.46 

26 

ERROR 

6270.55309 

11 

570.05028 



CRE 

1493.22055 

2 

746.61027 

1.17 

27 

ERROR 

14054.78199 

22 

638.85373 



DCRE 

2367.16949 

2 

1183.58475 

1.03 

28 

ERROR 

25313.40472 

22 

1150.60931 



PRE 

229.98731 

1 

229.98731 

0.66 

29 

ERROR 

3804.47242 

11 

345.86113 



DPRE 

311.99628 

1 

311.99628 

0.25 

30 

ERROR 

13485.75572 

11 

1225.97779 



CPRE 

502.93462 

2 

251.46731 

0.25 

31 

ERROR 

21939.96394 

22 

997.27109 



DCPRE 

2974.66321 

2 

1487.33160 

1.91 

32 

ERROR 

17090.53678 

22 

776.84258 



0.7648 

0.5433 

0.6885 

0.1082 

0.7698 

0.5260 

0.5385 

0.2556 

0.1363 

0.5113 

0.3293 

0.3741 

0.4321 

0.6239 

0.7793 

0.1712 
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Alert Level 


SOURCE 


MEAN 

ERROR 

TYPE 

ERROR 


SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

43573.57025 

1 43573.57025 

157»32 

0.0000 

3043.43182 

11 

273.96353 


1123.90092 

2 

561.95046 

6.24 

0.0071 

1979.74356 

22 

89.98939 


Event Types 




SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

83767.53351 

1 

83737.56351 

334.48 

0.0000 

2754.86460 

11 

250.44224 



14084.75040 

8 

1740.59380 

10.04 

0.0000 

15427.93308 

88 

175.31742 




Advisory Events 


SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

31171.31535 

1 

31171.61565 

175.02 

0.0000 

1959.12870 

11 

178.10261 



6816.46689 

4 

1704.11672 

14.54 

0.0000 

5158.15535 

44 

117.23081 



Caution 

. Events 




SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

36729.21120 

1 

36729.21120 

157.32 

0.0000 

2530.02383 

11 

232.72971 



3100.05540 

n 

1550.02770 

4.82 

0.0183 

70/1.37130 

22 

321.42598 




Discrete Actions 


Primary Variables 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

5890.56250 

1 

5890.56250 

494 » 86 

0.0000 

1 

ERROR 

130.93750 

11 

11.90341 




DIS 

1.56250 

1 

1.56250 

0,20 

0.6617 

2 

ERROR 

85.02083 

11 

7.72917 




CON 

3.57292 

2 

1.78646 

0,54 

0.5898 

3 

ERROR 

72.67708 

22 

3.30350 




DC 

8.09375 

2 

4.04688 

0,52 

0.6005 

A 

ERROR 

170.57292 

22 

7.75331 




PRO 

68.06250 

1 

68.06250 

12,89 

0.0042 

5 

ERROR 

58.10417 

11 

5.28220 




DP 

11.67361 

1 

11.67361 

2,22 

0.1640 

6 

ERROR 

57.74306 

11 

5.24937 




CP 

37.07292 

2 

18.53646 

2,47 

0.1076 

1 

ERROR 

165.01042 

22 

7.50047 




DCP 

8.27431 

2 

4.13715 

1,17 

0.3279 

8 

ERROR 

77.55903 

22 

3.52541 




REP 

10.02778 

1 

10.02778 

0.54 

- 0.4785 

9 

ERROR- 

204.97222 

11 

18.63384 




DR 

9.00000 

1 

9.00000 

1,95 

0.1900 

10 

ERROR 

50.75000 

11 

4.61364 




CR 

0.60764 

2 

0.30382 

0,03 

0.9684 

11 

ERROR 

207.89236 

22 

9.44965 




DCR 

11.32292 

2 

5.66146 

0,82 

0.4524 

12 

ERROR 

151.42708 

22 

6.88305 




PR 

0.02778 

1 

0.02778 

0,01 

0.9156 

13 

ERROR 

25.97222 

11 

2.36111 




DPR 

12.25000 

1 

12.25000 

1,61 

0.2306 

14 

ERROR 

83.66667 

11 

7.60606 




CPR 

3.06597 

2 

1.53299 

0.27 

0.7667 

15 

ERROR 

125.43403 

22 

5.70155 




61 


0.1472 



DCPR 

22.07292 

2 

11.03646 

2.09 

16 

ERROR 

116.01042 


5.27320 



EVE 

1.00000 

1 

1.00000 

0.14 

17 

ERROR 

78.58333 

11 

7.14394 



DE 

16.00000 

1 

16.00000 

1.98 

18 

ERROR 

89.00000 

11 

8.09091 



CE 

15.32292 

2 

7.66146 

1.19 

19 

ERROR 

142.09375 

22 

6.45881 



DCE 

21.26042 

0 

10.63021 

2.43 

20 

ERROR 

96.23958 

22 

4.37453 



PE 

4.00000 

1 

4.00000 

0.42 

21 

ERROR 

104.08333 

11 

9.46212 



DPE 

12.25000 

1 

12.25000 

1.84 

22 

ERROR 

73.41667 

11 

6.67424 



CPE 

27.19792 

2 

13.59896 

2.48 

23 

ERROR 

120.71875 

22 

5.48722 



DCPE 

14.84375 

2 

7.42187 

1.53 

24 

ERROR 

106.48958 

00 

4.84044 



RE 

45.56250 

1 

45.56250 

6.94 

25 

ERROR 

72.18750 

11 

6.56250 



ORE 

10.56250 

1 

10.56250 

1.76 

26 

ERROR 

65.93750 

11 

5.99432 



CRE 

11.38542 

2 

5.69271 

0.67 

27 

ERROR 

188.11458 

22 

8.55066 



DCRE 

55.90625 

2 

27.95313 

3.26 

28 

ERROR 

188.84375 

22 

8.58381 



PRE 

0.06250 

1 

0.06250 

0.01 

29 

ERROR 

83.85417 

11 

7.62311 



DPRE 

1.56250 

1 

1.56250 

0.24 

30 

ERROR 

71.60417 

11 

6.50947 



CPRE 

3.96875 

2 

1.98438 

0.22 

31 

ERROR 

197.86458 

00 

8.99384 



DCPRE 

15.03125 

2 

7.51563 

1.29 

32 

ERROR 

128.05208 

22 

5.82055 



0.7154 

0.1«7:i 

0.3242 

0.1113 

0.5289 

0.2027 

0.1070 

0.2380 

0.0232 

0.2113 

0.5239 

0.0577 

0.9295 

0.6338 

0.8038 

0.2950 
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Alert Level 


SOURCE 

SUM OF DEGREES OF 
SQUARES FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 
1 ERROR 

434.49317 1 434.49317 

24.01495 11 2.18318 

199*02 

0.0000 

TYPE 
2 ERROR 

17,18649 2 

9,49742 22 

8 » 59324 
0*43170 

19.91 

0.0000 


Event Types 




SOURCE 

SUM OF DEGREES OF 
SQUARES RREEDCM 

HEAN 

EQUAht 

F 

TAIL 

FRODABILITY 

MEAN 
1 ERROR 

1228.916B2 1 1228.91882 

43.55907 ll 3.96555 

310.03 

0.0000 

TYRE 
2 ERROR 

201.12207 8 

149.29067 83 

25.14026 

1.69648 

14.82 

0.0000 


Advisory Events 



SOURCE 

SUM OF DEGREES OF 
SQUARES FREEDOM 

MEAN 

SQU.ARE 

F 

TAIL 

PROBABILITY 

MEAN 
1 ERROR 

488,47355 1 

35.86334 11 

488.47355 

3.26030 

149,82 

0.0000 

TYPE 
2 ERROR 

66.19286 4 

40.68601 44 

16.54822 

0.92468 

17.90 

0.0000 


Caution Events 



SOURCE 

SUM OF DEGREES OF 
SQUARES FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 

1 ERROR 

567.13835 1 

18.13127 11 

567.13835 

1,64830 

344.08 

0,0000 

TYPE 
2 ERROR 

98.00684 2 

80.65479 22 

49.00342 

3.66613 

13.37 

0.0002 
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Heading RMS Error 


Primary Variables 



SOURCE 

SUM OF 

DEGREES OF 

HEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

796 » 58006 

1 

796.58006 

1537»27 

0.0000 

1 

ERROR 

5^69996 

11 

0.51818 




DIS 

1.41115 

1 

1.41115 

7.64 

0.0184 

T 

ERROR 

2.03046 

11 

0.18459 




CON 

0.26546 

2 

0.13273 

0.88 

0.4280 

3 

ERROR 

3.30971 

22 

0.15044 




DC 

0.35406 

0 

i. 

0.17703 

0.60 

0.5599 

4 

ERROR 

6.53901 

22 

0.29723 




PRO 

0.31313 

1 

0.31313 

0.82 

0.3836 

5 

ERROR 

4.18331 

11 

0.38030 




DP 

0.39533 

1 

0.39533 

2.61 

0.1347 

6 

ERROR 

1.66796 

11 

0.15163 




CP 

0.45728 

2 

0.22864 

0.66 

0.5285 

7 

ERROR 

7.66037 

00 

0.34820 




DCP 

0.63432 

0 

L. 

0.31716 

1.70 

0.2057 

8 

ERROR 

4.10321 

22 

0.18651 




REP 

0.08483 

1 

0.08483 

0.38 

0.5479 

9 

ERROR 

2.42797 

11 

0.22072 




DR 

2.02232 

1 

2.02232 

6.09 

0.0312 

10 

ERROR 

3.65253 

11 

0.33205 




CR 

2.25065 

2 

1.12532 

6.62 

0.0056 

11 

ERROR 

3.74075 

22 

0.17003 




DCR 

0.15148 

2 

0.07574 

0.34 

0.7121 

12 

ERROR 

4.83129 

oo 

i'.L. 

0.21960 




PR 

0.37976 

1 

0.37976 

1.51 

0.2446 

13 

ERROR 

2.76430 

11 

0.25130 




DPR 

0.00029 

1 

0.00029 

0.00 

0.9655 

14 

ERROR 

1.63559 

11 

0.14869 




CPR 

0.37671 

2 

0.18835 

0.53 

0.5973 

15 

ERROR 

7.85362 

22 

0.35698 
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0.0418 



DCPR 

1.26901 

9 

0.63451 

3.59 

16 

ERROR 

3.89158 

22 

0.17689 



EVE 

0.89855 

1 

0.89855 

3.38 

17 

ERROR 

2.92009 

11 

0.26546 



DE 

0.45282 

1 

0.45282 

1.40 

18 

ERROR 

3,55501 

11 

0.32318 



CE 

0.56185 

2 

0.28092 

1.15 

19 

ERROR 

5.38349 

22 

0,24470 



DCE 

0.37178 

2 

0.18589 

1.31 

20 

ERROR 

3.12634 

22 

0.14211 



F'E 

0.14919 

1 

0.14919 

0.53 

21 

ERROR 

3.07379 

11 

0,27944 



DPE 

0.78988 

1 

0.78988 

4.33 

22 

ERROR 

2.00484 

11 

0,18226 



CPE 

0.19854 

2 

0,09927 

1.34 

23 

ERROR 

1.63074 

22 

0.07412 



nCPE 

0.19560 

9 

0.09780 

0.63 

24 

ERROR 

3.39498 

22 

0.15432 



RE 

1.06004 

1 

1.06004 

5.42 

25 

ERROR 

2.15298 

11 

0,19573 



DRE 

0.00439 

1 

0.00439 

0.04 

26 

ERROR 

1.32193 

11 

0,12018 



CRE 

0,18763 

9 

0.09381 

0.45 

27 

ERROR 

4.53682 

22 

0.20622 



DCRE 

2.17994 

n 

i. 

1.08997 

4.73 

28 

ERROR 

5.06783 

22 

0.23036 



PRE 

1.24044 

1 

1.24044 

6.26 

29 

ERROR 

2.17902 

11 

0.19809 



nPRE 

0.00184 

1 

0.00184 

0.01 

30 

ERROR 

2.30648 

11 

0.20968 



CPRE 

0.35000 

2 

0.17500 

0.84 

31 

ERROR 

4,58972 

22 

0.20862 



DCPRE 

0.45619 


0.22810 

1.12 

32 

ERROR 

4.49793 

22 

0.20445 



0.0929 

0.2615 

0.3356 

0.2906 

0.4802 

0.0615 

0.2826 

0.5400 

0.0401 

0.8519 

0.6403 

0.0195 

0.0294 

0.9270 

0.4456 

0,3455 
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Alert Level 


SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 


SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

MEAN 

43.50941 

1 

43.50941 

950.15 

0.0000 

ERROR 

0.50371 

11 

0.04579 



TYPE 

0.3/985 

2 

0.18993 

2.97 

0.0724 

ERROR 

1.40905 

oo 

0.06405 





Event Types 




SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 


_ SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

MEAN 

128.67039 

1 

128.67039 

1030)09 

0.0000 

ERROR 

1.37403 

11 

0.12491 



TYPE 

0.40295 

8 

0.05037 

1.16 

0.3306 

ERROR 

3.81114 

88 

0.04331 
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Vertical Speed RMS Error 

Primary Variables 



SOURCE 

SUN OF 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


HEAN 

25682.59960 

1 

25682.59960 

504.33 

0.0000 

1 

ERROR 

560.16745 

11 

50.92431 




DIS 

9.15819 

1 

9.15B19 

1.05 

0.3273 

2 

ERROR 

95.85491 

11 

8.71408 




CON 

32.96426 

2 

16.48213 

1.29 

0,2966 

3 

ERROR 

282.16402 

22 

12.82564 




DC 

48.95214 

2 

24.47607 

1.14 

0.3379 

4 

ERROR 

4/2.19964 

22 

21.46362 




PRO 

9.32030 

1 

9.32030 

0.40 

0.5379 

5 

ERROR 

253.64652 

11 

23.05877 




DP 

18.37908 

1 

18.37908 

1.67 

0,2229 

& 

ERROR 

121.16724 

11 

11.01520 




CP 

41.45183 

2 

20.72592 

0.94 

0.4040 

7 

ERROR 

482.60437 

22 

21.93656 




DCP 

19.74835 

2 

9.87418 

0.99 

0.3879 

8 

ERROR 

219.63985 

22 

9.98363 




REP 

3.46425 

1 

3.46425 

0.25 

0.6273 

9 

ERROR 

152.74725 

11 

13.88611 




DR 

36.60754 

1 

36.60754 

2.30 

0.1579 

10 

ERROR 

175.33650 

11 

15.93968 




CR 

2.82336 

2 

1.41168 

0.12 

0.8899 

11 

ERROR 

264.94808 

22 

12.04309 




DCR 

6.86327 

2 

3.43163 

0,39 

0.6802 

12 

ERROR 

192.46242 

22 

8.74829 




PR 

27.24405 

1 

27.24405 

1.65 

0.2251 

13 

ERROR 

181.42872 

11 

16.49352 




DPR 

0.40481 

1 

0.40481 

0.15 

0.7054 

14 

ERROR 

29.57874 

11 

2.68898 




CPR 

0.75415 

2 

0.37708 

0.04 

0.9649 

15 

ERROR 

231.56365 

22 

10.52562 
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DCPR 

12.86391 

2 

6.43195 

0.41 

0.6710 

16 

ERROR 

348.25762 

22 

15.82989 




EVE 

135.90702 

1 

135.90702 

4.23 

0.0642 

17 

ERROR 

353.36541 

11 

32.12113 




DE 

10.27470 

1 

10.27470 

1.12 

0.3127 

18 

ERROR 

100.95108 

11 

9.17737 




CE 

8.18424 

2 

4.09212 

0.69 

0.5143 

19 

ERROR 

131.34540 

22 

5.97025 




DCE 

8.03007 

2 

4.01503 

0.53 

0.5949 

20 

ERROR 

166.07343 

22 

7.54879 




PE 

0.19988 

1 

0.19988 

0.01 

0.9065 

21 

ERROR 

152.07818 

11 

13.82529 




OPE 

13.62533 

1 

13.62533 

1.77 

0.2103 

22 

ERROR 

84.66280 

11 

7.69662 




CPE 

2.21231 

2 

1.10616 

0.08 

0.9228 

23 

ERROR 

301.72016 

22 

13.71455 




DCPE 

36.33055 

2 

18.16528 

1.46 

0.2542 

24 

ERROR 

273.99195 

22 

12.45418 




RE 

2.71288 

1 

2.71288 

0.31 

0.5902 

25 

ERROR 

96.96172 

11 

8.81470 




DRE 

27.70706 

1 

27.70706 

1.41 

0.2602 

26 

ERROR 

216.27776 

11 

19.66161 




CRE 

2.07231 

2 

1.03616 

0.10 

0.9012 

27 

ERROR 

218.12780 

22 

9.91490 




DCRE 

3.65869 

2 

1.82934 

0.14 

0.8663 

28 

ERROR 

278.58751 

22 

12.66307 




PRE 

0.02933 

1 

0.02933 

0.00 

0.9518 

29 

ERROR 

34.41414 

11 

7.67401 




DPRE 

6.58564 

1 

6.58564 

0.47 

0.5055 

30 

ERROR 

152.86153 

11 

13.89650 




CPRE 

14.68658 

2 

7.34329 

1.02 

0.3763 

31 

ERROR 

158.03852 

22 

7.18357 




DCPRE 

7.39940 

2 

3.69970 

0.40 

0.6729 

32 

ERROR 

201.75530 

22 

9.17070 
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Alert Level 



SOURCE 

SUH OF 

DEGREES OF 

HEAN 

F 

TAIL 



SQUARES 

FREEH0H 

SQUARE 


PROBABILITY 


MEAN 

2688 » 21509 

1 

2688.21509 

118.71 

0.0000 

1 

ERROR 

249.09430 

11 

22.64494 




^YPE 

28.42152 

2 

14.21076 

5.17 

0.0145 

0 

4 

ERROR 

60.50009 

22 

2.75000 




Event Types 



SOURCE 

SUH OF 

DEGREES OF 

HEAN 

F 

TAIL 


SQUARES 

FREEDOH 

SQUARE 


PROBABILITY 


HEAN 

6419.50022 

1 

6419.50022 

126,18 

0.0000 

1 

ERROR 

559.64953 

11 

50.87723 




TYPE 

195.02563 

8 

24.37820 

3.53 

0.0014 

T 

ERROR 

606.99455 

88 

6.89767 




Advisory Events 



SOURCE 

SUH OF 

DEGREES OF 

HEAN 

F 

TAIL 



SQUARES 

FREEDOH 

SQUARE 


PROBABILITY 


HEAN 

488.47355 

1 

488.4/355 

149.82 

0.0000 

1 

ERROR 

35.86334 

11 

3.26030 




TYPE 

66.19286 

4 

16.54822 

17.90 

0.0000 


ERROR 

40.68601 

44 

0.92468 




Caution Events 



SOURCE 

SUH OF 

DEGREES OF 

HEAN 

F 

TAIL 



SQUARES 

FREEDOH 

SQUARE 


PROBABILITY 


MEAN 

567.13835 

1 

567,13835 

344.08 

0.0000 

1 

ERROR 

18.13127 

11 

1.64830 




TYPE 

98.00684 


49.00342 

13.37 

0.0002 

n 

ERROR 

80.65479 

22 

3.66613 
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Airspeed RMS Error 


Primary Variables 



SOURCE 

suH or 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


HEAN 

1201.83557 

1 

1201.83557 

178.08 

0.0000 

1 

ERROR 

74.23933 

11 

6.74903 




DIS 

0.04375 

1 

0.04375 

0.06 

0.8165 

0 

ERROR 

8.52280 

11 

0.77480 




CON 

2.11318 

2 

1.05659 

3.31 

0.0552 

3 

ERROR 

7.01534 

22 

0.31888 




PC 

3.27016 

0 

1.63508 

1.27 

0.3009 

4 

ERROR 

28.35016 

22 

1.28864 




PRO 

1.46208 

1 

1.46208 

•0.85 

0.3751 

5 

ERROR 

18.82035 

11 

1.71094 




DP 

0.02176 

1 

0.02176 

0.05 

0.8229 

6 

ERROR 

4.55489 

11 

0.41408 




CP 

0.20154 

T 

0.10077 

0.07 

0.9335 

7 

ERROR 

32.12351 

22 

1.46016 




DCP 

5.80568 


2.90284 

3.97 

0.0336 

8 

ERROR 

16.07457 

22 

0.73066 




REP 

0.21545 

1 

0.21545 

0.50 

0.4961 

9 

ERROR 

4.78171 

11 

0.43470 




DR 

7.52131 

1 

7.52131 

8.19 

0.0155 

10 

ERROR 

10.10472 

11 

0.91861 




CR 

1.65765 

2 

0.82882 

1.40 

0.2677 

11 

ERROR 

13.02252 

22 

0.59193 




DCR 

1.24452 

2 

0.62226 

0.94 

0.4066 

12 

ERROR 

14.59665 

n 

0.66348 




PR 

2.93266 

1 

2.93266 

5.24 

0.0428 

13 

ERROR 

6.15385 

11 

0.55944 




DPR 

0.47266 

1 

0.47266 

1.11 

0.3147 

14 

ERROR 

4.68386 

11 

0.42581 




CPR 

1.13164 

2 

0.56582 

0.83 

0.4510 

15 

ERROR 

15.07240 

22 

0.68511 




0.1325 



DCPR 

3.85237 

2 

1.92619 

2.22 

16 

ERROR 

19.09899 

22 

0.86814 



EVE 

5.47950 

1 

5.47950 

5.15 

17 

ERROR 

11.71283 

11 

1.06480 



DE 

0.20175 

1 

0.20175 

0.21 

18 

ERROR 

10.54824 

11 

0.95893 



CE 

1.60775 

2 

0.80388 

0.88 

19 

ERROR 

20.06459 

22 

0.91203 



DCE 

2.40319 

2 

1.20159 

2.20 

20 

ERROR 

11.99896 

22 

0.54541 



PE 

1 191592 

1 

1.91592 

2.38 

21 

ERROR 

8.86902 

11 

0.80627 



DPE 

0.01085 

1 

0.01085 

0.03 

O') 

ERROR 

3.71710 

11 

0.33792 



CPE 

0.85599 

2 

0.42799 

0.94 

23 

ERROR 

10.06033 

22 

0.45729 



DCPE 

1.44772 

2 

0.72386 

1.74 

24 

ERROR 

9.15190 

22 

0.41600 



RE 

0.04803 

1 

0.04803 

0.11 

25 

ERROR 

4.76545 

11 

0.43322 



DRE 

0.18562 

1 

0.18562 

0.41 

26 

ERROR 

4.93730 

11 

0.44885 



CRE 

2.68285 

2 

1.34143 

1.99 

27 

ERROR 

14.81221 

22 

0.67328 



DCRE 

2.51851 

2 

1.25926 

2.43 

28 

ERROR 

11.42139 

22 

0.51915 



PRE 

0.26095 

1 

0.26095 

0.90 

29 

ERROR 

3.18895 

11 

0.28990 



DPRE 

1.54795 

1 

1.54795 

2.35 

30 

ERROR 

7.23980 

11 

0.65816 



CPRE 

0.20455 

2 

0.10228 

0.17 

31 

ERROR 

13.34022 

22 

0.60637 



DCPRE 

2.19987 

2 

1.09993 

1.69 

32 

ERROR 

14.34309 

22 

0.65196 



0.0444 

0.6554 

0.4283 

0.1342 

0.1515 

0.8610 

0.4073 

0.1988 

0.7454 

0.5334 

0.1602 

0.1117 

0.3631 

0.1534 

0.8459 

0.2081 
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Alert Level 


SUH OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

?4. 02211 

1 

94.02211 

314.00 

0,0000 

3.29372 

11 

0.29943 



0.27210 

2 

0.13605 

1.28 

0.2972 

2.33355 

22 

0.10607 



Event Types 



SUK OF 

DEGREES OF 

MEAN 

F 

TAIL 

SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 

243.2H68 

1 

248.21468 

240.75 

0.0000 

1L34116 

11 

1.03101 



1.87075 

8 

0,23384 

1.24 

0.2837 

16.54810 

88 

0,18805 




I 

I 

I 
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Control Activity 

Primary Variables 



SOURCE 

SUM OF 

DEGREES OF 

HEAR 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

406512.50694 

1 

406512.50694 

212.65 

0,0000 

1 

ERROR 

21028.61806 

11 

1911.69255 




DIS 

35.00694 

1 

35.00694 

0.14 

0.7169 

2 

ERROR 

2781.61806 

11 

252.87437 




CON 

760.01389 

2 

380.00694 

0.70 

0.5060 

3 

ERROR 

11696.61111 

22 

540.75505 




DC 

951.26389 

9 

475.63194 

2.17 

0.1376 

A 

ERROR 

4814.61111 

22 

218.84596 




PRO 

1201.77778 

1 

1201.77778 

1.39 

0.2630 

5 

ERROR 

9499,26389 

11 

863.56944 




DP 

240.25000 

1 

240.25000 

1,05 

0.3266 

6 

ERROR 

2506.45833 

11 

227.85985 




CP 

122.59732 

2 

61.29861 

0.20 

0.8165 

7 

ERROR 

6591.36111 

22 

299.60732 




DCP 

75.54167 

0 

X. 

37.77083 

0.12 

0.8849 

8 

ERROR 

6755,50000 

22 

307.06818 




REP 

6.25000 

1 

6.25000 

0.03 

0.8620 

9 

ERROR 

2170.79167 

11 

197.34470 




DR 

152.11111 

1 

152.11111 

0.57 

0.4677 

10 

ERROR 

2956.93056 

11 

268.81187 




CR 

141.29167 

2 

70.64583 

0,30 

0,7465 

11 

ERROR 

5245.16667 

22 

238.41667 




DCR 

437.59722 

2 

218.79861 

0,82 

0.4546 

12 

ERROR 

5889.61111 

22 

267.70960 




PR 

82.50694 

1 

82.50694 

1.34 

0.2714 

13 

ERROR 

676.78472 

11 

61.52588 




DPR 

10.56250 

1 

10.56250 

0.03 

0.8589 

H 

ERROR 

3506.89583 

11 

318.80871 




CPR 

179.18056 

0 

89.59028 

0.76 

0.4810 

15 

ERROR 

2604.27778 

22 

118.37626 
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DCPR 

285.79167 

2 

142»89583 

0*73 

0*4953 

16 

ERROR 

4333.25000 

22 

196*96591 




EVE 

6765.06250 

1 

6765*06250 

40*21 

0.0001 

17 

ERROR 

1850.47917 

11 

168*22538 




DE 

8.50694 

1 

8*50694 

0*05 

0*8194 

18 

ERROR 

1711.70139 

11 

155*60922 




CE 

348.29167 

9 

174.14583 

2.13 

0.1422 

19 

ERROR 

1794.91667 

22 

81.58712 




DCE 

178.84722 

2 

89*42361 

0*78 

0.4723 

20 

ERROR 

2533.94444 


115*17929 




PE 

23.36111 

1 

23*36111 

0.12 

0.7325 

21 

ERROR 

2091.09722 

11 

190*09975 




DPE 

90.25000 

1 

90.25000 

0.51 

0.4B94 

OT 

ERROR 

1941.37500 

11 

176*48864 




CPE 

553.93056 

2 

276*96528 

1.42 

0.2619 

23 

ERROR 

4276.11111 

22 

194*36869 




DCPE 

289.04167 

0 

i. 

144*52083 

1*21 

0.3167 

24 

ERROR 

2623.08333 

22 

119.23106 




RE 

205.44444 

1 

205,44444 

1.93 

0.1919 

25 

ERROR 

1169.18056 

11 

106*28914 




DRE 

121.00000 

1 

121.00000 

1*02 

0,3340 

26 

ERROR 

1303.79167 

11 

118*52652 




CRE 

94.59722 

7 

47*29861 

0.33 

0.7191 

27 

ERROR 

3103.77778 

22 

141.30808 




DCRE 

187.79167 

2 

93.89583 

0.84 

0.4439 

28 

ERROR 

2450.66667 

22 

111*39394 




PRE 

351.56250 

1 

351*56250 

2,17 

0.1690 

29 

ERROR 

1783,97917 

11 

162*17992 




DPRE 

52.56250 

1 

52*56250 

0.36 

0.5616 

30 

ERROR 

1614.31250 

11 

146*75568 




CPRE 

613.04167 

2 

306.52083 

2*03 

0.1552 

31 

ERROR 

3322.16667 

22 

151*00758 




DCPRE 

489.12500 

7 

244,56250 

1.44 

0*2573 

32 

ERROR 

3724.00000 

22 

169*27273 
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Alert Level 



SOURCE 

SUH OF 

DEGREES OF 

MEAN 

F 

TAIL 


SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

28419.77843 

1 

28419.77843 

197.20 

0.0000 

1 

ERROR 

1585.30410 

11 

144.11855 




TYPE 

369.70502 

9 

184.85251 

16.78 

0.0000 

2 

ERROR 

242.38149 


11.01734 




Events Types 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

FAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILITY 


MEAN 

72575.17169 

1 

72575.17169 

18i. 76 

0.0000 

1 

ERROR 

Am. 27779 

11 

399.29798 




TYPE 

439.72965 

8 

54.96621 

1.88 

0.0728 

T 

ERROR 

2570.16373 

88 

29.20641 




Advisory Events 



SOURCE 

SUM OF 
SQUARES 

DEGREES OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

1 

MEAN 

ERROR 

27036.62211 

1515.22932 

1 

11 

27036.62211 

137.74812 

196.28 

0.0000 

o 

i. 

TYPE 

ERROR 

44.92951 

561.01057 

2 

22 

22.46475 

25.50048 

0.88 

0.4285 



Caution Events 




SOURCE 

SUM OF 
SQUARES 

DEGREES OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

1 

MEAN 

ERROR 

39511.57617 

2689.00859 

1 

11 

39511.57617 

244.45533 

161.63 

0.0000 

A. 

TYPE 

ERROR 

193.31135 

1642.95183 

4 

44 

48.32784 

37.33982 

1.29 

0.2869 
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Workload Rating 



SOURCE 

SUM OF 

DEGREES OF 

MEAN 

F 

TAIL 



SQUARES 

FREEDOM 

SQUARE 


PROBABILI1 


HEAN 

9004^82000 

1 

yOOI. 82000 

180.46 

0.0000 

1 

ERROR 

548.88667 

11 

49.89879 




DIS 

59.40500 

1 

59.40500 

13.06 

0.0041 

'} 

ERROR 

50.03500 

11 

4.54864 




CON 

28.95083 

2 

14.47542 

17.92 

0.0000 

3 

ERROR 

17.77250 

22 

0.80784 




DC 

44.06033 

2 

22.03042 

7.02 

0.0044 

4 

ERROR 

69.02917 

22 

3.13769 




PRO 

27.38000 

1 

27.38000 

2.56 

0.1380 

5 

ERROR 

117.72667 

11 

10.70242 




DP 

14.04500 

1 

14.04500 

2.90 

0.1164 

6 

ERROR 

53.19500 

11 

4.83591 




CP 

0.75250 

2 

0.37625 

0.08 

0.9250 

7 

ERROR 

105.87083 


4.81231 




DCP 

5.64250 


2.82125 

0.99 

0.3867 

8 

ERROR 

62.54750 


2.84307 




REP 

10.88889 

1 

10.88889 

4.45 

0.0585 

9 

ERROR 

26.89111 

11 

2.44465 




DR 

1.22722 

1 

1.22722 

0.77 

0.4001 

10 

ERROR 

17.61944 

11 

1.60177 




CR 

1.66028 

o 

0.83014 

0.69 

0.5136 

11 

ERROR 

26.58972 

oo 

i-i. 

1.20862 




DCR 

6.26028 

2 

3.13014 

1.68 

0.2097 

12 

ERROR 

41.02306 

no 

i-i. 

1.86468 




PR 

2.13556 

1 

2.13556 

1.86 

0.2001 

13 

ERROR 

12.64444 

11 

1.14949 




DPR 

0.12500 

1 

0.12500 

0.05 

0.8200 

14 

ERROR 

25.32167 

11 

2.30197 




CPR 

7.40194 

o 

A. 

3.70097 

1.64 

0.2162 

15 

ERROR 

49.54806 

22 

2.25218 




DCPR 

0.49083 

2 

0.24542 

0.11 

0.8971 

16 

ERROR 

49.49250 

22 

2.24966 
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